SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning
Authors: Yangruibo Ding
Jinjun Peng
Marcus J. Min
Gail Kaiser
Junfeng Yang
Baishakhi Ray
Venue: NeurIPS 2024 poster (2024)
Keywords: Code Language Models
Program Semantics Learning
Code Execution Reasoning
Primary area: generative_models
TL;DR: This paper introduces SemCoder, a novel 6.7B Code LM trained with comprehensive program semantics from static code to dynamic execution, achieving superior performance in code generation and execution reasoning.

Abstract: Code Large Language Models (Code LLMs) have excelled at tasks like code completion but often miss deeper semantics such as execution effects and dynamic states. This paper aims to bridge the gap between Code LLMs' reliance on static text data and the need for semantic understanding for complex tasks like debugging and program repair. We introduce a novel strategy, _monologue reasoning_, to train Code LLMs to reason comprehensive semantics, encompassing high-level functional descriptions, local execution effects of individual statements, and overall input/output behavior, thereby linking static code text with dynamic execution states.
We begin by collecting PyX, a clean Python corpus of fully executable code samples with functional descriptions and test cases. 
We propose training Code LLMs not only to write code but also to understand code semantics by reasoning about key properties, constraints, and execution behaviors using natural language, mimicking human verbal debugging, i.e., rubber-duck debugging. This approach led to the development of SemCoder, a Code LLM with only 6.7B parameters, which shows competitive performance with GPT-3.5-turbo on code generation and execution reasoning tasks. SemCoder achieves 79.3% on HumanEval (GPT-3.5-turbo: 76.8%), 63.6% on CRUXEval-I (GPT-3.5-turbo: 50.3%), and 63.9% on CRUXEval-O (GPT-3.5-turbo: 59.0%). We also study the effectiveness of SemCoder's monologue-style execution reasoning compared to concrete scratchpad reasoning, showing that our approach integrates semantics from multiple dimensions more smoothly. Finally, we demonstrate the potential of applying learned semantics to improve Code LLMs' debugging and self-refining capabilities. Our data, code, and models are available at: https://github.com/ARiSE-Lab/SemCoder.

SEMCODER: Training Code Language Models with Comprehensive Semantics Reasoning
Yangruibo Ding
Jinjun Peng Columbia University
Marcus Min Columbia University jm50250columbia edu
Columbia University yrbding@cs. cowumbia.....ooooooo..o........a edu
j jinjunecs.coluee columbia. edu
Gail Kaiser
Junfeng Yang
Baishakhi Ray Columbia University rayb0cs columbia edu
Columbia University kaiserEcs. columbia. edu
Columbia University junf engeen. columbia. edu
Abstract
Code Large Language Models (Code LLMs) have excelled at tasks like code com- pletion but often miss deeper semantics such execution effeets and dynamic states. This paper aims to bridge the gap between Code I.LMs' reliance on static text data and the need for semantic understanding for complex tasks like debugging and program repair. We introduce a novel strategy, monologue reasoning, to train Code LLMs to reason comprehensive semanties, encompassing high-level functional de- scriptions, local execution effects of individual statements, and overall input/output behavior, thereby linking static code with dynamic execution states. We begin by collecting PyX, a clean Python corpus of fully executable code samples with functional descriptions and test cases. We propose training Code LLMs not only write code but also to understand code semantics by reasoning about key properties, constraints, and execution behaviors using natural language. mimicking human ver- bal debugging, i.e., rubber-duck debugging. This approach led to the development of SEMCODER, Code LLM with only 6.7B parameters, which shows competitive performance with GPT-3.5-turbo on code generation and execution reasoning tasks. SEMCODER achieves 79.3% on HumanEval (GPT-3.5-turoo: 76.8%), 63.6% on CRUXEval-1 (GPT-3.5-turbo: 50.3%), and 63.9% on CRUXEval-O (GPT-3.5- turbo: 59 99.... We also study the effectiveness of SEMCODER'S monologue-style execution reasoning compared to concrete scratchpad reasoning, showing that approach integrates semantics from multiple dimensions more smoothly. Finally, demonstrate the potential of applying learned semantics to improve Code LLMs' debugging and self-refining capabilities. Our data, code, and models are available com/ Att///RSEE--EE Lab/ Laae--LLb//Seeodrr /Semooder
Introduction
have revolutionized the field of programming (6-8). These models, trained primarily on vast corpora of programming-related text such source code and docstrings 9 excel automating tasks like code generation.
Recent advancements in code language models (Code LMs) 1+55
Unfortunately. the reliance on static text data limits the ability of existing Code LMs to understand what the programs actually doing, especially to reason about the deeper semantics intrinsic to code execution. The lack of semantic understanding unsurprisingly often leads to poor performance in debugging and repairing errors in generated code [10]. Code LMs struggle with reasoning about program semantics in both static and dynamic settings. In a static setting, the challenge lies in understanding the intended behavior of the code without running requiring deep comprehension of
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

code syntax and static semantic properties (e.g., program dependency graph, etc.) [11 A dynamic setting involves observing and interpreting the code's behavior during execution, including tracking variable changes, identifying runtime errors, and detecting performance issues [3] Even when the execution traces are exposed to the model, [33 observed that Code LMS could not effectively interact with the real executions, struggling to leverage the dynamic execution traces for debugging.
Fifty years ago, Terry Winograd envisioned the future Al programmer: "The key to future pro- gramming lies in systems which understand what they are doing (14)". In this paper, we explore constructing such a programming system, backed up by language models, not only to write programs but also to understand what they are doing (a.k.a., semantics). Our key insight is that Code LMs should mimic how pragmatic human developers work: starting with general specifications, breaking them down into sub-tasks with expected properties and constraints, implementing code line by line while reasoning about the effects of each line, and checking overall correctness by examining execution effects [15. To achieve this, we introduce a novel strategy to train Code LMs to reason comprehensive program semantics.
We train SEMCODER, a novel semantic-aware Code LM. We incorporate different modalities of program semantics: (i) High-Level Functional Descriptions: We train SEMCODER to understand high-level functional descriptions bi-directionally by both generating code from natural language and summarizing code as natural language. This involves teaching models to grasp program's s purpose, akin to how a human developer outlines software high-level approximate semanties; (ii) Key Properties and Constraints: we train SEMCODER extract the functional properties and constraints of program, which should hold for all scenarios and corner cases. (iii) Overall Execution Behavior: we train SEMCOdeR to understand the local impact of individual code statements, recognizing how each line affects variables, control flow, and memory usage. By grasping these effects, models can better predict code execution semantics. We train the model to learn both abstract and concrete semantics, teaching it the general purpose of statement and illustrating it with concrete examples.
Curating Executable Code Dataset We collect PYX, synthetic dataset capturing comprehensive program semantics with executable code samples and unit tests. Inspired by existing datasets [I6] 17). we use powerful LLM to synthesize NL-to-code pairs. To ensure quality, PyX includes only executable samples. It also generates unit tests and detailed execution traces, recording program states after each statement. From PYX, we further construct a debugging dataset, PYX-R. PyX-R includes buggy code snippets generated by Code LMs, corresponding debugging rationales, and refine plans 13 leading to patches. By fine-tuning Code LMs on PYX-R, we aim develop programming assistants that debug and patch faulty code in human-like manner, advancing the capabilities of current Code LMs in iterative programming.
Learning Program Semantics To learn program semantics, we propose monologue reasoning: Code LMs try to understand and explain the code semantics to themselves. Code LMs will summarize the program funetionalities, highlight the key properties and constraints, and reason code execution step-by-step. inspired by rubberduce debugging [1]. The code execution reasoning will be performed in two directions: (i) forward monologue: SEMCODER uses source code and inputs to verbally simulate execution, explaining each line's impact, executed lines, variable changes, and final output, and (ii) backward monologue: given the final output, SEMCODeR reasons about possible previous states abstractly, capturing essential characteristics without precise enumeration. This abstract reasoning is crucial for understanding complex operations like sorting or aggregation, where the previous state cannot be uniquely determined. Overall, monologue reasoning equips Code LMs with a human-like understanding of control flow, state transitions, and complex operations, bridging the gap between static code analysis and dynamic exccution reasoning.
We show that, by training on this approach, SEMCODER can generate, reason about execution, debug and refine code in a more intuitive and effective manner, pushing the boundaries of what current Code LMs can achieve in different software engineering tasks.
Performance of SEMCODER SEMCODER, while having only 6.7B parameters, exhibits excep- tional performance in code generation and execution reasoning tasks, surpassing larger models like GPT-3.5-turbe and various open-source models. For code generation, SEMCODER variants achieve a pass @1 I of 79.3% on HumanEval (, outperforming GPT-3.5-turbo's 's 76.8%, and with 27.5% on LivehodeBench-LLie (18), outperforming GPT-3.5-turbo's 's 23.9%. For execution rea- soning, SEMCODER variants score 63.6%, 65.1% 1%, 61.2% on CRUXEval-L, CRUXEval-O, and LiveCodeBench- Cow.enchc--ooexeettiooo respectively, significantly outperforming baseline models including
2

---

Approximate Semantics
Symbolic Semantics
Implement function takes patential energies, sorts them in escending order, removes duplicates, and returns the indices of the unique sorted
from typing inport List def unique_ souuuasorrdd___ indices (erarçies: list(float)):
energy diet - 1]
energies.
for idx, energy in enumerate lenerg-est
energy_diet (ttpjditt.ettt att..aatdfaaltteeeeee
idx)
Write function unique sorted, indioes(energies: List(floct] - List(lnt] that takes a list of petential energies as input and neturns list of indices corresponding unique sorted energles.
sorted unniuuniiieeeeee energies eorted tenergiest)
[energy_ cietttttteeee [eneray] for energy in socted unique energims]
unique sosttoorrodddtee
return unique_sorten www__ssttt_s__see Andices
Q Abstract Semantics
Operational Semantics
Given the input [10.5, 8.2, 10.5, 8.2"" _.2'",ee''s simulate the execution
Key Properties and Constraints
Filling Dictionary: The 'orr' loop begins, iterating through the with "erumerate": For
"eeenyyddcc" initalized
dictionary.
imput can contain duplicate and variable length.
'energy: = 10.5" We update the dictionary to (10.5: 0)
-Iter For = 4 'energy = 8.2", Since 8.2 the dictionary, it is not updated.
- The output list will have unique energy values sorted in
Findl state of "energy_ dict" : [10.5: 0, 8.2: 1, 7.1: 3)"
ascending order und will contain their original indices
Serting Unique Energies: Resulting in sorred_ wwwpuuuuiuueeeeeeeeeeeeeeeeeeeee energies [7.1, 8.2, 10.5]"
from the irput
Retrieving driginal Indices:
an energy value repeated, only first eccurrence's as index stored the dictionory. The function ensures that the indices in the output list
For energy 7.1" the index is
For enengy: 8.2", the index "1"""
For eenery: 10.9" the index is "O".
neflect the order of the unique volued of hor sorting.
Thus, "unique_sorted" indices' becomes 1,
Return Statement: The function returns the list 13,
Figure I: SEMCODER'S 's training strategy with different modalities of program semantics. We specify the overall objective of a task, i.e., the approximate semanties (blue box), such as "retrieves potential energies of atoms and performs sorting" followed by the corresponding code solution (pink box). Then we annotate the abstract code semantics as those key properties and constraints (red box) that hold regardless of inputs. Beyond static semantics, we also pair code with test cases, such as "Given [10.5, 8.2, 10.5, 7.1, 8.2], return [3. 1, 0]". We further annotate the dynamic, operational semantics with forward and backward monologues (yellow box, and more in Section 4.2). SeMCoder learns from all the information to not only generate code but comprehensively reason its semantics.
GPT-3.5-turbo and showcasing its superior understanding of program executions. The innovative monologue reasoning technique, where the model verbalizes code semantics from high-level function- alities to low-level execution details, greatly enhances execution reasoning, outperforming existing trace reasoning formats like scratchpad [2] and NExT 13.. The monologue reasoning approach also allows SEMCODER to flexibly handle abstract semantics and non-deterministic program states, which existing methods struggle with. Additionally, SEMCODER excels in debugging and self-refinement, improving code generation accuracy iteratively by verbally rubber-duck debugging by itself without the need for dynamic tracing. We empirically reveal that SEMCODER 's static monologue reasoning is comparably effective as attaching real traces 13] for bug fixing. Besides the effectiveness, monologue reasoning has unique advantages by design: (1) it is purely static reasoning and does not require dynamic tracing, (2) it compacts the execution reasoning by focusing on key properties related to the bug rather than checking all redundant program states and concrete variable values, and (3) it provides a human-readable explanation for better understanding.
Our main contribution is the development of SEMCoDER, semantic- Code LM designed to enhance understanding and reasoning about program semantics. We introduce Monologue Reasoning, a novel code reasoning approach that connects static source code with its runtime behavior through detailed verbal descriptions of code properties and runtime behaviors. To expose comprehensive program semantics at different levels, we curate PyX, collection of executable code samples with functional descriptions and execution traces. SEMCODER demonstrates superior performance in code generation and execution reasoning tasks, surpassing larger open-source models. SEMCODER also excels in debugging and self-refinement by leveraging knowledge from its semantic-aware training. Our work highlights the potential of integrating deep semantic understanding into Code LMs to improve their effectiveness in complex programming tasks.
2 Program Semanties
Program semantics refers to the meaning or behavior of computer program, describing what it does when it runs, including input processing, computations, and output [19 20). Understanding program semantics is crucial for ensuring programs behave correctly and meet their intended purpose.
3

---

Program semantics can be represented in various modalities, A high-level description outlines program's 's intended functionalityy while fine- fine-graine grained semantics detail the actions and side effects of each line of code, including data manipulation and state changes. This detailed understanding helps developers write better code and aids code reviewing, debugging, and team communication. Fine-grained semantics can be concrete or abstract. Concrete semantics (e.g.. program traces) capture actual execution effects, while abstract semantics focus on key input-output relationships and overall program effects, abstracting away lower-level details [21 [22]. Following the existing literature on program semantics 19 20]. we curate the following semantics.
Approximate Semantics describes the overall objectives of a program, often articulated through docstrings or documentation [23] These Natural Language descriptions provide an overview of the program's 's goals and anticipated results, ensuring that the implementation aligns with the intended high-level functionalities (blue box in Figure [
Symbolic Semantics represents complex functionality and logic in a way that both humans and machines can interpret consistently. It refers the layer of meaning derived from the symbols, syntax, and structure of source code (pink box in Figure D. It describes how code represents high- level functionality and logic by focusing on those constructs within the source code that symbolize particular behaviors, concepts, or operations in the program design.
Operational Semanties describe how the individual steps in source code execute 25 19 20] It
focuses on describing the concrete execution of a program in a step-by-step manner, detailing how each action transforms the program's state. This approach is particularly useful for reasoning about the dynamic behavior of programming languages (yellow box in Figure D
Abstract Semantics is way to describe program behavior at higher level of abstraction 22, 27
[2] Unlike concrete semantics, which provides detailed descriptions the program's execution on specific inputs, abstract semantics focuses on the essential aspects of program behavior while ignoring low-level details. This approach is to reason about program properties and constraints (red box in Figure I that always hold.
3 PyX: Semantic-aware Training Dataset
Capturing program semantics requires executing source code with unit tests. Real-world datasets are challenging due to diverse configurations. lack of unit tests, and limited documentation (28). Thus, we use a synthetic dataset to capture program semantics. Here, we detail the process of gathering high-quality data for learning multi-modal code semantics. Similar to 16 17]. we first synthesize NL to Code pairs. Then, we use the Python interpreter to filter out defective samples, ensuring comprehensive semantic coverage. See Appendix for more details and analysis, including Figure which depicts the data collection procedure.
3.1 Synthesizing Executable Code
Synthesizing instructional data (NL to code) with existing LLMs is common for obtaining large datasets for instruction tuning CodeLLMs [16 17 However, current methods do not guarantee the quality of generated code. For instance, out of 13.1k Python solutions from [16], about 11.6k 9%) are inexecutable despite instructions to produce "correct" and "self-contained" code (Table 7 in Appendix shows the top 10 error types). To build SEMCODER, we train it only with executable data, as good data leads to better generation 22 33 We improve the ISS-INSTRUCT data generation process [16], which prompts LLM to create programming task and solution inspired by a seed snippet. Instead of randomly sampling lines from existing programs, we parse them into ASTs and sample subtrees to obtain parsable seeds. We execute the generated code, retaining only successfully executed samples, and use the generator model's debugging capability to retry until the code runs correctly. With the low-cost supervision from the Python interpreter, we build higher-quality instruction tuning dataset for semantic-aware model training. Step of Figure 4 in Appendix [F] summarizes this process. Table 2 in Appendix | compares our PyX with OSS-INSTRUCT in details.
3.2
Dataset with Operational Semantics
We select a subset of PYX to construct data to learn the execution reasoning (See Step-Il of Figure in Appendix F)

---

Forward Monologue
Backward Monologue
Given the imput energhes [30.5, 0., 10.5, 71, 8.2] what elll the function retur??
13, 1, "unction insur?
with Dietlenary: The
-enenyy diet' letrolzedd as on empmy dictiseery. begite, Iterating through
the 1hird "O"
Kay canstraints according to axtoott
energy cornusponds
"1",
Ther 1: For 'ide O'. "anergy 10 5", We updete the dictisnary to (10.5
Enerav Volues:
enerw
'eneroles11"" second unisse Det's )=z')
"enerdeetor" third uniue enerww fet'ss call It "43)
Iter Si Fer 'ide 4', "anargy 6.2" Since 5.2 is the dictionary. la not updolted.
In the sorted let urique energies. el mutt be the analect. followed by
"enerev "210.5: B.2: 7.3: aV" Serting Energles: Resslting 10, 61"
simple nut values array con created. We murt exarr that should The energy at index "s' is less than the energy et inder 'I" and both less the energy index
(7.1,
Retrievise Original Irdices:
Than, unique sorted, inalices becomes
For enerry + 30.5 the index is '0"
2" the Indexx '1"
.enersiesf3) 1.0" (fisst unioae smallest value)
"enensiesto: indexing repetition,
.enenjess1] 3.0 (sooadd usieee. middle value)
deplicates of euch voluee for inshance: energias [3.0, 5.0, 5.0.1.0.00 3.0g
Figure 2: Forward monologue simulates the execution step-by-step, and backward monologue deduces the previous program states by making assumptions and checking with observed constraints.
Data Selection We apply the following filtering criteria to select programs with clean execution flow from our executable dataset: (i) Only programs without external resource interactions (e.g., keyboard input, file system changes) are included, as our trace representation only captures variable state changes. (ii) Programs must have no randomness, ensuring predictable behavior.
Input Generation Our executable dataset typically has one or two example inputs per program. To model operational semanties accurately and avoid bias, we need diverse input set to expose different execution traces. We expand the input set using type-aware mutation and LLMM-ased input generation, similar to 31 as detailed in Appendix [F]
3.3 PyX-R: Training Code LLMs to Rubber-duck Debug and Self-refine
We construct debugging dataset, PyX-R, to train Code LLMs for debugging and self-refinement, aiming to improve their iterative programming capabilities. We collect buggy solutions by sampling LLM for problems in PYX and keep those responses that fail at least one of the tests. We perform rejection sampling with LLM to collect rubber-duck debugging rationales for buggy programs and their input sets. PyX-R only includes those rationales that lead correct patches, verified by differential testing against the ground truth. We provide an example of PyX-R data in Appendix
SEMCODER: Learning Comprehensive Semantics
4.1 Natural Language to Code
We train SEMCODER translate high-level functional descriptions into executable code, known as the natural language to code task [66 17). Using PYX samples, we provide well-defined problem descriptions that specify (1) the task's overall objective, (2) implementation constraints, and (3) expected outcomes with test cases. These descriptions give a holistic view of the task, forming the basis for the model's understanding.
4.2 Monologue Reasoning to Comprehensively Understand Code Semantics
We train SEMCODER to understand code semantics through monologue reasoning: Given the source code and executable inputs/outputs, the model needs to reason code from high-level abstraction to low-level details, from static perspective to dynamic perspective. Note that the original natural language description of the problem will not be provided to generate monologues.
First, SEMCODER summarizes the high-level functionalities to understand the approximate semantics. Then, SEMCODER will explain the abstract semantics as key properties and constraints that always hold for all executions. Finally, SEMCODER describes the operational semantics by articulating state changes during execution for the provided execution input/output. Inspired by rubber-duck debugging, this approach explains program states transition more smoothly than structured formats like Scratchpad [32), avoiding redundant program states (e.g., numpy array with hundreds of elements) and concrete values (e.g., float numbers) while focusing on key properties that contribute to the code understanding. We detail such effectiveness Section 6.2. We provide partial monologues for
illustration in Figure 2 and full monologues in Appedix G
5

---

4.2.1 Forward Monologue
We provide SEMCODER with the source code and input, and it learns to reason the operational semantics by verbally simulating the execution step by step and predicting the execution output (Tigure]yelllon box).
Execution Coverage
To ensure comprehensive understanding, forward monologue covers those lines with side effects, contributing to thorough control flow understanding and enforcing a detailed code walkthrough, similar to a developer's debugging process.
To mimic natural code execution, forward monologue follows the natural order of reasoning. For loops, it explains each iteration with specific values, addressing lines executed multiple times differently. This ensures an accurate, context-aware execution path, similar how developers mentally simulate execution behavior, helping detect issues like infinite loops or incorrect condition handling.
Natural Execution Orders
Understanding code side effects is crucial for grasping program state evolution. Forward monologue indicates changes in variable values when a line is executed, enhancing its ability to simulate real execution effects. This focus side effects helps capture dynamic semantics, providing granular, step-by-ster explanations of state changes, thus improving debugging and refinement based observed behavior.
Program State Transition
Final Output Finally, the model predicts the program's final output after explaining the execution process to validate the correctness of intermediate logic.
4.2.2 Backward Monologue
While forward execution is mostly deterministic, the previous program state cannot always be determined from the current state, such unsorted list from its sorted version. Therefore, design the backward monologue to be flexibly abstract (See Figure blue box).
Abstract Intermediate Constraints In our backward monologue reasoning, we use abstract intermediate constraints when previous program states can't 't be uniquely determined from the current state, such as after sorting aggregation. We train the model to describe these constraints abstractly. This abstraction captures essential characteristics and patterns, allowing the model to reason about multiple possible previous states. This approach enhances the model's flexibility and generalization, improving its ability to handle diverse and complex program reasoning tasks.
For a given output, the model learns to predict concrete input values that satisfy the input abstract constraints. This step bridges the gap between abstract reasoning and concrete execution. This ensures it understands patterns and can generate practical examples, enhancing its robustness for real-world tasks like debugging and testing. This capability mirrors how human developers perform backward reasoning for debugging [3
Concrete Input
4.2.3 Monologue Annotation Using LLM
To annotate the monologue required for training SEMCODER, we employ a method of rejection sampling [3 35 through large language model. We leverage the power of LLM to automatically annotate numerous samples for training SEMCODER, while we have an execution- based golden standard verify the quality of annotated monologues, ensuring they are informative and valuable, thereby enhancing SEMCODER's 's ability to reason about program executions both forward and backward.
For forward monologue annotation, we feed code samples from our PyX dataset into an LLM, prompting it to generate detailed explanation of state changes and transition logic, ending with final output prediction. We then execute the code; if the actual output matches the LLM's prediction, we accept the monologue, ensuring it accurately reflects the program's 's execution. If the output does not match, the monologue is rejected. This method ensures the monologue is comprehensive and suitable for training SEMCODER. We follow similar strategy for backward monologue annotation.
To enhance our monologue annotation process, we provide the LLM with few-shot examples when generating forward and backward monologues. These examples follow our defined rules, explicitly detailing execution lines, variable changes, and reasoning steps for forward monologues, and abstract constraints with specific examples for backward monologues. This guidance ensures the LLM adheres

---

to our structured reasoning steps. We also use system instructions to ensure the LLM follows the procedures illustrated in the few-shot examples.
4.3 Joint Training with Comprehensive Semantics
SEMCODER trained with the combined data of natural- lanerral-anggagettoo samples, forward mono- logues, and backward monologues, using the standard next-token prediction objective [36]. Our training has an emphasis on learning the program semantics, where the training loss is accumulated only by cross-entropy loss on code and monologue tokens together. We also include task-specific prefix as part of the model input that the model is better aware of which types of program semantics it should learn to capture and predict for the current sample. See Appendix H for concrete prefixes.
5
Experiments
Code Generation and Execution Reasoning For code generation evaluation, we consider EvalPlus 33 and the code generation task in LiveCodeBench-LLit (LCB-Lite for (18) For execution reasoning, we employ CRUXEval [37 and the code execution task LiveCodeBench (LCB-Exec for short) [8). We prompt the baseline models to perform chain-of-thouge reasoning [38 motivated by two-shot examples, and zero-shot prompt SEMCODER to perform monologue reasoning, Inferences all follow the benchmark's 's original settings.
Rubber-duck Debugging and Self-refine We evaluate iterative programming capabilities a setting similar to self self-refiiement/seee wewpe///eeeeeebgginngg [39 40)
-models generate code, test it, rubber-duck debug the erroneous solution, and refine their code based on the root cause analysis. Using EvalPlus [31, we perform five iterative refinements using greedy decoding. We evaluate models with both zero-shot prompting and fine-tuned using PyX-R settings.
Models SEMCODER loads the 6.7B base version of DeepSeekCoder as the initial checkpoint and continues to optimize with the proposed program semantic training. Similar Magicoder [16). we train two versions of SEMCODER. the base version and the more advanced SEMCODER-S The base version of SEMCODER is completely trained with PYX. The advanced SEMCODER-S1 is trained with an extended dataset that includes PyX, Evol-instruct [16]. and partial CodeContest [41]
Evol-instruch is decontaminated version of evol-codealpaca- v1 [42] which contains numerous cerruction-follwwiin data. To increase the diversity of coding problems, we sample solutions from CodeContest resulting in 4.3k problems with at least one correct, -.L--renerated solution.
Configuration and Empirically Settings All SEMCODER variants are trained for 2 epochs on server with eight NVIDIA RTX A6000 GPUs, using learning rate of 5e-5 with cosine decay to 5e-6 during the program semantics training. For self-refinement fine-tuning, SEMCODER and baseline Code LLMs are trained for 2 epochs with learning rate of le-5. We a batch size of 512, a maximum context length of 2,048. Similar to [6], we use GPT-3. 5-turbo to synthesize coding problems. To minimize the cost, we use -miin to generate code solution and monologue reasoning texts, which are typically longer sequences than the problem descriptions.
Evaluation
6.1 Overall Performance
In this section, we report the overall performance of SEMCODER for code generation and execution reasoning tasks and compare it with baseline Code L.MSs.
Baselines and Evaluation Metric We consider four families of open-source Code LLMs as baselines: Code Llama 44) StarCoder2 [5], DeepSeekCoder [3] and Magicoder [66. Despite SEMCODER having only 6.7B parameters, include 6.7B. 7B, and 13B variants, both base and instruct versions, if publicly available, totaling 13 open- source models. We also compare SEMCODER to GPT-3.5-uun t-trbb for code generation and execution reasoning to measure the performance gap with closed-soures models. Results are reported with @
CEMCODER Achieves Dominant Performance in Code Generation and Execution Reasoning We show the main evaluation results in Table T SEMCODER reports dominant performance in exccution reasoning, significantly better than other open-source baselines, including those with
7

---

Table 1: Overall performance of SEMCODER. For code generation, the numbers outside and inside parenthesis *)" indicate the base and plus versions of EvalPlus, respectively. All results are reported with pass @ 1. CXEval indicates CRUXEval, and LCB indicates LiveCodeBench.
Code Generation
Execution Reasoning
Model
Size
HEval (+)
MBPP (+)
LCB-LLite
CXEval-1
CXEval-O
L.C--Exec
GPT-3.5
76.8 (70.7)
82.5 (69.7)
23.9
50.3
59.0
43.6
CodeLlama-
13B
42.7 (38.4)
63.5 (52.6)
10.6
40.5
36.0
23.2
Codel Lama-Inst
13B
49.4 (41.5)
63.5 (53.4)
12.5
45.6
41.2
25.7
StarCoder2
15B
46.3 (37.8)
55.1 (46.1)
16.0
46.9
46.2
33.6
StarCoder2 Inst
15B
67.7 (60.4)
78.0 (65.1)
15.5
47.1
50.9
29.6
Codel Laaaa- Python
7B
37.8 (35.4)
59.5 (46.8)
7.1
40.4
34.0
23.0
Codel CodeLIlmm-Inn Jama-Inst
7B
(31.1)
56.1 (46.6)
10.6
36.0
36.8
30.7
StarCoder2
7B
35.4
54.4 (45.6)
11.6
38.2
34.5
26.3
Magicoder-Cl.
7B
60.4 (55.5)
64.2 (52.6)
11.4
34.0
35.5
28.6
Magicoder-S S-CCL
7B
70.7 (67.7)
68.4 (56.6)
12.1
42.0
35.8
30.0
DeepSeekCoder
6.7B
47.6 (39.6)
72.0 (58.7)
20.3
39.5
36.1
DeepSeckCodderInstt
6.7B
73.8 (70.7)
74.9 (65.6)
21.1
41.9
43.2
34.0
Magicoder- DS
6.7B
66.5 (60.4)
75.4 (61.9)
25.5
45.5
41.9
38.8
Magicoder-S-DD
6.7B
76.8 (71.3)
75.7 (64.4)
23.3
44.6
43.5
38.4
SEMCODER (Ours)
6.7B
73.2 (68.9)
79.9 (65.3)
22.4
62.5
65.1
59.7
SEMCODER-S (Ours)
6.7B
79.3 (74.4)
79.6 (68.5)
27.5
63.6
63.9
61.2
2x more parameters. We also collect results for larger models (e.g., CodeLlama-34B) from the benchmark to compare with SEMCODER in Appendix Table
Comparing SEMCODER with its initial checkpoint, DeepSeckCoder-6.
semantic-heard train- ing strategy brings much stronger execution reasoning capabilities, resulting in a 23.0% absolute improvement for input prediction and 23.9% and 23.6% absolute improvement for CRUXEval-O and LCB-Exee, respectively. Notably, both variants of SEMCODER outperform GPT-3.5-turbo for execution reasoning with significant margin.
SEMCODER also demonstrates remarkable performance code generation: SEMCODER achieves 79.9 pass@ @ in MBPP, outperforming all open-source baselines, and the advanced version SEM- CODER-S achieves pass@) of 79.3 and 74.4 for HumanEval base and plus, respectively, significantly beating other models, including GPT-3.5-turbo. These impressive results support Terry Winograd's vision in 1973 [14] that training models to thoroughly understand programs produces more reliable and accurate programming assistants.
Execution Reasoning Requires Comprehensive Understanding Code Semantics We show re- sults of input/output prediction without reasoning in Appendix Table Interestingly, when comparing the results with reasoning vs. w/o reasoning, we found that the free-form chain-of-thought can hardly help model reason about execution, even if it takes more inference-tine time computation generate more tokens. In contrast, monologue reasoning significantly improves the execution reasoning capability by up to 21.7% absolute improvement in output prediction. This empirically reveals that thorough understanding of code execution requires systematic reasoning over comprehensive semantics.
6.2 Effectivenss of Monologue Reasoning
In this section, we perform ablation studies to demonstrate the effectiveness of monologue reasoning. Baselines We consider two bascline execution reasoning approaches: scratchpad 2 and NeXT's trace format [3]. NeXT adds numeric order to state changes and omits intermediate loop states. We also create template to concise execution traces, replacing monologue reasoning with concrete program states. Examples are in Appendix[]] Additionally, we report few-shot prompting results on the base Code LM using chain-of-thought reasoning 38
without execution reasoning data.
Experiments We first construct different formats of execution reasoning using the same PyX samples that construct monologues. Then we fine-tune ddepseek-coder. 7b-base on these

---

Table 2: Ablation study for input and output prediction with different types of execution reasoning.
Method
CRUXEval-I CRUXEval-O
LCB-Exec
Few-shot Prompting
39.5
41.2
36.1
Finetune
w/ Scratchpad [
48.8
50.6
39.9
w/ NeXT 13
49.4
50.9
32.2
w/ Concise Trace
52.1
55.6
35.9
w/ Monologue Reasoning (Ours)
61.8
63.5
58.5
different execution reasoning data for epochs and compare their results on input and output prediction using CRUXEval.
Monologue Reasoning is More Effective Than Learning Concrete Program States Results in Table show show that, while all baselines improve execution reasoning, our monologue reasoning outperforms them in input and output prediction with clear margins. The main reason is that monologues describe state transitions smoothly natural language while keeping track of only key properties and values, which is casier for code LLMs to learn and understand and consequently enhance execution reasoning. In contrast, baselines provide only concrete states with redundant information and values while not explaining the causal relations of these transitions, so code LLMs struggle to capture the correlation among them.
When we manually check the monologues, which are structured to ensure correct outcomes (Section 4.2.3 we observe that the intermediate logic could be occasionally flawed - the model sometimes makes wrong assumptions about code properties but still reaches the correct result. In contrast, all baselines are guaranteed to have correct intermediate steps, as they are realistic execution traces (See Appendix A for limitation and future work). Empirically, however, the model learns more effectively from the monologues. This highlights the potential benefits of emphasizing key property correctness and model- friendlly data format when jointly training code LLMs with distinct semantics.
6.3 Debugging and Self-Refinement
We format the debugging process as verbally and statically explaining why the bug happens [T5] evaluate the code LMs' reasoning capability rather than the tool-using capability that performs dynamic exccution with tracers or debuggers. Then the model should fix the bug according to its own reasoning, i.e., self-refine. We provide an example in Appendix (Example-2) illustrate how this task is performed.
We consider four state-of-the-ar instruction crunne code LMs baselines: Alama-3.1 3.1- Instruct-8B [43], DeepSeekCoder ConrSseecCodrr--tstureee 7B, Magicoder- DS-6.7B and Magicoder-S-DD DS-6.7B. We evaluate their static debug and self-refine capabilities on EvalPlus with five iterations. We first evaluate with zero-shot prompting and then also fine-tune with PYX-R to illustrate its value.
Experiments
Table Performance of iterative debug and self-refine
(a) Zero-shot Prompting
(b) Fine-tuned PyX-R
Self-Refine
Self-Refine
Model
Model
HEval (+)
MBPP (+)
HEval (+)
MBPP (+)
Magicoder-D DS
65.2 (60.4)
78.3 (65.9)
Magicoder-DS Magicoder-$ 5-D
78.8 (64.3)
83.1 (66.7)
"Magicoder- 5DDS
77.4 (70.1)
79.9 (68.8)
83.5 (76.2)
(4.4 .4)
DeepSeekCoder--nng
77.4 (73.2)
80.4 (69.6)
DeepSeckCCode--nnss
83.5 (75.6)
(69.6)
Llama-3. 3.1 Insst
76.8 (68.9)
77.8 (65.6)
Llama-3, I Inst
76.8 (68.9)
76.7 (61.4)
SEMCODER
75.6 (71.3)
83.1 (67.2)
SEMCODER
76.8 (69.5)
81.7 (65.9)
SEMCODER-S
84.8 (79.3)
86.8 (74.3)
SEMCODER-S
85.4 (79.3)
87.0 (73.5)
SEMCODER Reports Promising Performance in Debugging and Self- Refinement In Table SEMCODER-S outperforms all baselines, more notably in the zero-shot setting. This result illustrates that the SEMCODER'S 's monologue reasoning augments general-purpose instruction tuning with code semantics reasoning capabilities. Appendix D demonstrates SEMCODER's 's continuous code

---

refinement throughout iterations, showcasing the potential of learned program semantics for complex programming tasks.
PyX-R Improves Iterative Programming Capability Fine-tuning Code LMs on PYX-R signifi- cantly improves iterative programming performance due to the monologue-style debugging rationale and well-aligned patches. PYX-R helps Code LMs understand and analyze bugs from source code and execution traces, aiming to inspire better iterative programming capabilities. We notice that PYX-R provides limited improvement to SEMCODER variants and Llama-3. I-Ins, and we speculate that these models are already trained with high-quality reasoning. and the occasional errors PyX-R debugging rationale restrict these models from becoming significantly better (See Appendix AA)
Monologue Reasoning vs. Execution Traces for Debugging We perform additional experiments by replacing the monologue reasoning part (See "### Execution Simulation" in Appendix Example 2) in the debugging rationale with real traces, following the format of NExT (13] and fine-tuning
code LMs again. Results are in Appendix C.T We notice that monologue reasoning is comparably effective as attaching execution traces. Besides the effectiveness, monologue reasoning has unique advantages by design: (1) it is purely static reasoning and does not require dynamic tracing, (2) it compacts the execution reasoning by focusing on key properties related to the bug rather than checking all redundant program states and concrete variable values, and (3) it provides human-readable explanation for better understanding.
7 Related Work
Code LLMs and Training Data Many open source Code LLMs, such as CodeGen 44 StarCoder [5) Code Llama [4) and DeepSeck Coder [3), are proposed. Specialized models [32 have also been developed for tasks like code generation, summarization, output prediction, and competitive programming following the success of GPT-3 [46.. These models are trained only on source code and related text, lacking execution context. This limits their understanding of program semantics, leading to security issues and debugging failures. We aim to bridge this gap by training Code LMs on both static source code and dynamic execution traces. An orthogonal line of research curates synthetic instruction-folloiin data to enhance Code LLM performance. Code Alpaca 47 has 20k instruction- response pairs, Wyol-Instruet-Code [17 expands this to 80k pairs, and DSS-Instruce [16] includes 75k diverse pairs from the Stack dataset However, these datasets focus on lanuurll-anguag-tto-ooooo tasks with little coverage of code execution and unverified solutions. To improve correctness, Zheng et al. [48) created a multi-turn conversation dataset with compiler error messages, and Wei et al. [49] incorporated execution by generating test cases and filtering invalid pairs. Yet, no dataset includes simulating and understanding execution traces. We aim to fill this gap (see Section 3
Learning and Reasoning about Program Executions Before LLMs, 50 51 predict simple program outputs using RNNs, GNNs, small transformers, and neural Turing machines. Austin et al. 32 fine-tuned LLMs for execution output prediction with minimal performance gains. Early models predicted final outputs without revealing execution traces. Nye et al. 2 introduced the Scratchpad method for intermediate results, and others [52) 53) fine-tuned UniXcoder (54) for execution traces
but didn't it evaluate for code generation tasks. We fine-tune a Code LLM to understand program semantics, excelling in code generation, output prediction, and input prediction (see Section Another approach uses execution feedback for debugging Code LLMs. Self-Debuggin [39] shows that natural language explanations unit test results help seff-refinement, but execution traces reduce performance. LeTI (55] and CYCLE 40 tine-tun with execution feedback to improve performance, especially for smaller models. NExT B3
generates debugging rationales to mitigate the negative impact of execution traces. Our work shows that a model trained on code generation, output prediction, and input prediction excels in understanding execution feedback and self-refinement (see Table 3
8 Conclusion
We train SEMCODER to simultancously learn different modalities of program semantics: Approx- imate, Symbolic, Operational, and Abstract. We show that such semantics-oriented joint training cultivates a comprehensive understanding of program semantics - SEMCODER or SEMCODER-S achieves SOTA performance, among all less-than-155 open-source models, in not only the code generation and input/output prediction but also tasks that require deep knowledge of both source code and execution execution reasoning like debugging and self-refinement.
10

---

Acknowledgement
This work was supported part by. DARPA/NIWC-Paciit N66001-21-C-4011 multple Google Cyber NYC awards. an Columbia SEAS/EVPR Stimulus award, NSF CNS-1845995, CNS-2247370, CCF-222 1943, CCF-23 CCF-1845893, and CCF-2107405. Any opinions, findings, conclu- sions, or recommendations expressed herein are those of the authors and do not necessarily reflect those of DARPA, or NSF.
References
[1] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan. Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Hcidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov. Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code, 2021.
[2] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models, 2021.
[3] Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseck-coder. When the large language model meets programming the rise of code intelligence, 2024.
[4] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre. Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2024.
[5] Anton Lozhkov Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Noua- mane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang. Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li. Wen-Ding Li. Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh. Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder 2 and the stack v2: The next generation, 2024.
[6] GitHub. Github copilot: Your ai pair programmer. https://gittms http:://ggtccbb.co.ffeatutees
copilot 2021.
[7] Amazon. Amazon codewhisperer: Your pi-powered productivity tool for the ide and command line. http:///wwweaaaaaoooooooooo/a//a aaaaazon.cco/oooddessepeeerrr 2022.
[8] OpenAl. Chatgpt. htttss:/chattptt.cm
2022.
[9] Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al. The stack; th of permissively licensed source code. arXiv preprint arXiv:2211.15533, 2022.
11

---

[10] Alex Gu, Wen-Ding Li, Naman Jain, Theo X. Olausson, Celine Lee, Koushik Sen, and Armando Solar-Lezama. The counterfeit conundrum: Can code language models grasp the nuances of their incorrect generations?, 2024.
[11] Nathaniel Ayewah, William Pugh, David Hovemeyer, David Morgenthaler, and John Penix. Using static analysis to find bugs. IEEE Software, 25(5):22-29 2008
[12] Brittany Johnson, Yoonki Song, Emerson Murphy-Hin Hill, and Robert Bowdidge. Why don't t software developers static analysis tools to find bugs? In 2013 35th International Conference on Software Engineering (ICSE), pages 672-681,20 2013.
[13] Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinlin Deng. Kensen Shi, Charles Sutton, and Pengcheng Yin. Next: Teaching large language models to reason about code execution, 2024.
[14] Terry Winograd. Breaking the complexity barrier again. In Proceedings of the 1973 Meeting on Programming Languages and Information Retrieval, SIGPLAN '73, page 13-30, New York, NY, USA, 1973, Association for Computing Machinery.
[15] Andrew Hunt and David Thomas. The pragmatic programmer: from journeyman to master. Addison- Adelson-Wesley Longman Publishing Co., Inc., USA, 2000.
[16] Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code all you need. arXiv preprint arXiv:2112.02122, 2023.
[17] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma. Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint #Xiv:2306.08566 2023.
[18] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024.
[19] Matthew Hennessy. The semantics of programming languages: an elementary introduction using structural operational semantics. John Wiley & Sons, Inc., 1990.
[20] Glynn Winskel. The formal semantics of programming languages: an introduction. MIT press, 1993.
[21] Carl A Gunter. Semantics of programming languages: structures and techniques. MIT press, 1992.
[22] Joseph E Stoy. Denotational semantics: the Scott-Strachey aphroccc to programming theory, MIT press, 1981.
language
[23] Steve McConnell. Code complete. Pearson Education, 2004.
[24] David Thomas and Andrew Hunt. The Pragmatic Programmer: your journey to mastery. Addison- Addison-Wesley Professional, 2019.
[25] Gordon D Plotkin. A structural approach operational semanties. Aarhus university, 1981.
[26] Flemming Nielson, Hanne R Nielson, and Chris Hankin. Principles of program analysis. springer, 2015.
[27] Patrick Cousot and Radhia Cousot. Abstract interpretation: unified lattice model for static analysis of programs by construction or approximation of fixpoints. In Proceedings of the 4th ACM SIGACT-SIGPLA symposium on Principles of programming languages, pages 238 252, 1977.
[28] Marcus J Min, Yangruibo Ding, Luca Buratti, Saurabh Pujar, Gail Kaiser, Suman Jana, and Baishakhi Ray. Beyond accuracy: Evaluating self-consistency of code llms. In The Twelfth International Conference on Learning Representations, 2023.
12

---

[29] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorn,, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann. Gustavo de Rosa, Olli Saarikivi, al. Textbooks are all you need. arXiv preprint arXiv:2306.1 1164, 2023.
[30] Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander J Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large language model as attributed training data generator: A tale of diversity and bias. Advances in Neural Information Processing Systems, 36, 2024.
[31] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024.
[32] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. ArXiv preprint, M0//2108.07332 2021.
[33] Marcel Böhme, Ezekiel O Soremekun, Sudipta Chattopadhyay, Emamurho Ugherughe, and Andreas Zeller. Where the bug and how is it fixed? an experiment with practitioners. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ES- EC/FSE 2017, page 117-128, New York, NY, USA, 2017. Association for Computing Machin- ery.
[34] George Casella, Christian P. Robert, and Martin T. Wells. Generalized accept-reject sampling schemes. Lecture Motos-Monoggaph Series, 45:342-347, 2004.
[35] Radford M. Neal. Slice sampling, 2000.
[36] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodci, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl preprint, 2019.
[37] Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint 2tt:::200000006.. 2024.
[38] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.
[39] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug, 2023.
[40] Yangruibo Ding, Marcus J. Min, Gail Kaiser, and Baishakhi Ray. Cycle: Learning self-refine the code generation, 2024.
[41] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser. Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition- code generation with alphacode. ArXiv preprint, abs/2203.07814, 2022.
[42]
theblackcat102. The evolved code alpaca dataset. (abpaaaccccct11//2//oooooeaeeeee
2023.
[43] Meta Llama Team. The llama herd of models, 2024.
[44]
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In International Conference on Learning Representations, pages 1-25, 2023.
[45] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et Starcoder: may the source be with you! arXiv preprint arXiv:2305.066 2tt.222305.066... 2023.
13

---

[46] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, pages 1-25, 2020.
[47] Sahil Chaudhary. Code alpaca: An fistructionnfoolooinng llama model for code generation. sahi hctpc///aaa11200111/coaapppaaaa 2023.
[48] Tianyu Zheng, Ge Zhang, Tianhao Shen, Xucling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinteepreeer:: Integrating code generation with execution and refinement, 2024.
[49] Yuxiang Wei, Federico Cassano, Jiawei Liu, Yifeng Ding, Naman Jain, Harm de Vries, Leandro von Werra, Arjun Guha, and Lingming Zhang. Starcoder2-instrual Fully transparent and permissive self-alignment for code generation, 2024.
[50] Wojciech Zaremba and Ilya Sutskever. Learning to execute, 2015.
[51] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines, 2014.
[52] Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sundaresan, and Nan Duan. Code execution with pre-trained language models, 2023.
[53] Yangruibo Ding, Benjamin Steenhoek, Kexin Pei, Gail Kaiser, Wei Le, and Baishakhi Ray. Traced: Execution-aware pre-training for source code. In Proceedings of the 46th IREF/ACM International Conference on Software Engineering, pages 1-12, 2024.
[54] Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. Unixcoder: Unified cross- moosl pre-training for code representation. arXiv preprint Siti:2203.038555 2022.
[55] Xingyao Wang. Hao Peng. Reyhaneh Jabbarvand, and Heng Ji. Leti: Learning generate from textual interactions, 2024.
[56] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024.
[57] higoode-onjechhicoaeedusss
https://githe http:///ttttu..... com www.e//iiioddepooeeeeeeeeeeeeeee
bigcode- dataset, (Accessed on 05/17/2024).
14

---

A Limitations and Future Work
Process Supervision for Intermediate Reasoning Steps We manually review the monologues in PYX and the rubber-duck debugging rationales in PYX-R, which are structured to ensure correct outcomes (see Section 4.2.3). While the final answers are accurate, we observed that the intermediate reasoning steps are occasionally flawed. Sometimes, the model makes incorrect assumptions about code properties but still reaches the right result, i.e., correct execution input/output and correct patch. Conceptually, such subtle mistakes in the intermediate reasoning steps might have negative impacts on further improving models' code reasoning capability.
We encourage future work to propose automatic and efficient process supervision approaches [56] specifically for code reasoning, which will be useful to further improve the quality monologues in PYX.
Curation of Monologue Annotation The monologue annotation data (Section
is crucial for
SEMCODER to excel at the output prediction and input prediction tasks. However, we rely on a more powerful LLM, GPT-3.5-Turbo or GPT-4o-mini to generate these annotations and employ rejection sampling from its responses, since our base model is relatively small with 6.7B parameters.
We encourage future work to try our semantic-oriented joint training on larger base model, so that it will be possible to generate the monologue annotations using the base model itself like Ni et did to bootstrap high-quality reasoning for self-refinement.
Incorporating Execution Reasoning into Code Generation We demonstrate that training on input and output prediction tasks are indirectly beneficial for both natural- nntural-lnguaageto-rr downstream tasks like self-refinement. However, there a more direct way to further improve the performance in code generation and self-refinement we can ask the model to first self self-verify its own solution by generating forward monologue (Section 4.2.1) for the test cases given in the natural language specification before finalizing the solution.
generation and
We encourage future work explore the possibility of using model's own execution reasoning ability to directly assist its code generation and self-refinement process.
B Broader Impacts
Social Impact In this work, we train semantic-aware Code LMs. We make all the data, code, and model checkpoints available publicly. The artifact could be used to deploy automated programming assistants that improve the developers' productivity. It possible but unlikely that the Code LMs will generate buggy or wrong code, but we suggest to use our models as "copilot" to assist with human developers rather than completely relying on the model for full automation.
Safeguards Our data is synthesized using a commercial LLM, i.c., -.t-b.5-turbo, which has been aligned by the releasing company, OpenAI, to avoid leaking personal malicious information. We regard our data has minimal risk of being misused due to its synthetic instinct.
C More Evaluation Results
C.1 Debug and Self-refine w/ Real Execution Traces
Table 4: Debug and self-refine with real traces in the format of NExT [3
Relf-Refine
Model
HEval (+)
MBPP (+)
Magicoder-DS
72.0 (66.5)
83.3 (67.5)
Magicoder-5 S-DS
81.7 (74.4)
83.9 (72.0)
DeepSeek Inst
84.8 (79.9)
85.4 (70.4)
Llama-3.1
(69.5)
82.8 (66.7)
SEMCODER
78.0 (70.7)
83.6 (66.4)
SEMCODER-S
86.0 (80.5)
87.0 (73.3)
15

---

C.2
Input/Output Prediction Without Reasoning
In T
we present the results of direct prediction for execution input and output without reasoning.
Table 5: Performance of direct prediction for execution input/output w/o reasoning.
Execution Reasoning
Model
Size
CRUXEval-I
CRUXEvaL-O
LCB-Exee
GPrbb
49.0
49.4
39.2
CodeLlama Python
13B
38.5
39.7
36.1
CodeLlama-Inss:
13B
47.5
40.8
33.8
StarCoder2
15B
47.2
46.9
34.7
StarCoder2-Inst
15B
47.4
47.1
8.1
CodeLlama- Lythonnn-Python
7B
37.3
34.6
31.1
CodeLlama- Inst
son you
7B
34.8
35.6
30.1
StarCoder2
7B
34.2
35.6
34.0
Magicoder-CI.
7B
32.0
35.6
32.4
Magicoder-S-CI.
7B
36.2
34.8
30.5
DeepSeekCoder
6.7B
42.2
43.6
44.5
DeepSeekCoder-Inns
6.7B
34.9
40.8
41.1
Magicoder-DS
6.7B
41.2
43.4
38.4
Magicoder-S-DD
6.7B
42.1
44.4
39.2
SEMCODER (Ours)
6.7B
46.9
47.9
38.0
SEMCODER-S (Ours)
6.7B
47.6
46.6
40.7
C,3
Comparison with Larger Open-Sourced Models and Closed-Source Models
Code Generation
Execution Reasoning
Model
Size
HEval (+)
MBPP (+)
CRUXEval-I
CRUXEval-O
GPT-3.5-Turbo-1 106
76.8 (70.7)
82.5 (69.7)
49.0 50.3
49.4 59.0
Claude-3-Opus
82.9 (77.4)
89.4 (73.3)
64.2 / 73.4
65.8 82.0
GPT-4-0613
(79.3)
69.8 75.5
68.7/77.1
2024-04-09
90.2 (86.6)
68.5 / 75.7
67.7/82.0 82.0
CodeLlama
34B
51.8 (43.9)
69.3 (56.3)
/ 50.1
42.4/43.6
DeepSeckCoder
33B
51.2 (44.5)
46.5
48.6/-
DeepSeckCoddr-Insst
33B
81.1 (75.0)
80.4 (70.1)
46.5 /-
49.9
SEMCODER (Ours)
6.7B
68.3 (62.2)
79.9 (65.9)
51.2/
46.1/
SEMCODER-S (Ours)
6.7B
81.1 (76.2)
78.8 (66.9)
48.
54.5
44.9/ 54.1
Table 6: Overall performance of SEMCODER vs. other Code LLMs. For code generation, the numbers outside and inside parenthesis " ")" indicate the base and plus versions EvalPlus, respectively. For execution reasoning, the left side and the right side of the slash "/" indicate the direct prediction and prediction with reasoning, respectively. All results are reported with pass@1. 1.
D SEMCODER Continuously Refines Code Qualities
We studied SEMCODER's 's code generation accuracy at each step of refinement with varied tempera- tures. The results are plotted Figure We observed that SEMCODER capable of continuously refining its own errors, and the increase does not stop when the temperature is high, which indi- cates the SEMCODER has strong debugging and self-refine capabilities and high temperate better leverages such capabilities for iterative programming.
16

---

Self refine Performance on humaneval
Self-refine Performance on mbpp
greedy
08
greedy
Emg-0.2
Emg-0.2
Emp-0.8
tmp-0.8
85
Pess
Pass@1
83
Refine Steos
Refine Steos
HumanEval
(b) MBPP
Figure 3: SEMCODER-S" zero-shot performance of self-refinement at each time step with different sampling strategies.
E
Executability Analysis of OSS-INSTRUCT
Table 7: Top-10 error types of inexecutable Python code in OSS-Instruct [6]
Error
#Cases of 43.1k Python samples
Type
ModuleNolFounddEror
3417
NameError
1954
FileNotFoundError
1052
ImportError
979
EOFError
743
SyntaxError
672
IndentationError
506
ArtributeErroo
213
TypeError
196
ValueError
132
We execute all Python samples in OSS-INSTRUCT 16 to analyze their executability. To get more accurate result, try mitigate the MocuulelooFFunnErrroo by installing the top 75 missing dependencies according to a pre-run result. Table 7 shows the breakdown of the top 10 error types.
Details on PYX
The whole data collection pipeline is shown in Figure 4 Here also document more details about PYX.
Prompt for Data Synthesis We follow the prompt in OSS- INSTUUTT for data synthesis, but with two modifications: 1) For problem design, instruct model to avoid interaction with external resources or requirement of uncommon third-party libraries increase the probability of getting executable code. 2) For giving solutions, instruct the model to show its thought process before writing code to produce more aligned natural language along with the code in the dataset. Table 10 details our prompts with an example in PYX.
Input Set Expansion To enlarge the input set, we first initialize the input corpus with all known valid inputs. Then, for type-aware mutation, we alter known inputs based on type-specific heuristics. For LLM-based generation, we prompt the model with the function and several known inputs to generate more. We verify new inputs by executing them, retaining only those that execute successfully without exceptions. We alternate between type-aware mutation and LLM--baeed generation until reaching a predefined threshold, combining mutation's 's efficiency with L.MM generation's 's robustness. The generated inputs and their outputs serve unit tests for the NL-described task in future steps.
17

---

use Task Descripiio, Executable Code
Query
Weaker
Model
OSS-Inatruct
Data Simplification
Weak Solution
Data Synthesis
Existing Inputs Collection
Instruction
Evecuaaie
More Inputs Generation
Test Cases Execulion
Fail Q Trace
Retry with Feedbacks
Type-aware Mutation
Buggy Code w/
Generator Model
LLM-based Generation
Faulty Traces Feedback
Vertyy
Query
Generator
(Problem. Solution) Pairs
Environment
Model
Success
Pass]
Test Cases Execution
X Error
Execulion Environment
Task Descriplion, Single-function Code, Oracles
Faully Traces Feedback,
Buggy Code,
Parsting Execution Error
Step-l: Synthesizing Executable Code
Step-ll: Data for Execution Reasoning
Step-ill: PYX-R: Training Code LMe to debug and selt-refine
Figure 4: PyX: Execution-aware Training Data Collection Strategy
Table 8: The comparison between OSS-INSTRUCT and PyX.
Seed
Solution
Performance
Dataset
Problems
Parse
Parse
Execute
HumanEval (+)
MBPP (+)
OSS-INSTRUCE
75k
Partially
67.1 (61.0)
77.5 (64.3)
OSS- DOS-IINTTUCCT-PIEE Pythhn
43k
48%
97%
73%
66.5 (59.8)
78.6 (65.9)
PYX (Ours)
32k
100%
100%
100%
70.1 (64.0)
78.6 (65.1)
Coverage of Inputs Our input generation method only considers diversity in terms of variable values but does not try to fully exercise different execution paths executable code, like what the goverage-guidest testing usually does. However, our dataset only consists of relatively short single-function programs that do not have complicated branches. We find that our generated inputs can achieve average branch coverage and average line coverage of 93%, 96% respectively, which shows that our approach is light-weight yet effective for the current setting.
Executability v.s. Parsability Along with generation of 34k executable data, we also get 13k samples that can only be parsed as valid ASTs but cannot be successfully executed. We include these parsable samples to form 47k dataset, PYX-SOFT, and compare it against the fully executable one PyX. Results in Table show that PyX-SOFT and PyX have comparable results, and both of them outperform OSS-INSTRUCHS and its Python subset. Though PyX-SOFT sometimes slightly outperforms PYX, full exccutability is required for comprehensive semantic coverage, PYX for the main evaluation.
Data De-duplication We follow the data decontamination process of [16] and 57 to clean our dataset. To examine the similarity between our instruction tuning dataset and the testing benchmarks, we evaluate the "edit similarity" between them, specifically by scaling the metric fuzz. partial, aeeiiilltteeee._ae seckke_soottrraeeee
provided by thefuzz library. For each sample in our dataset, its similarity to a benchmark is computed the maximum cosine similarity to any sample in the benchmark. We apply the same analysis to similarity between majority has less than 0.4 similarity, which indicates that the performance improvement brought by dataset is not from data leakage benchmark data imitation.
for comparison. F shows that the dataset and the two benchmarks is par with OSS-INSTRUCT. where the
Categories To study the effect of executability filtering, we categorize all samples in our dataset following [16] shown by Figure Compared to OSS- the categorical distribution shifts by an increase algorithmic and data structure problems, data science and machine learning problems, and mathematical and computational problems, and a decrease in the remaining categories, which is expected since interactions with external resources commonly required in scenarios like database, web and UI design not allowed in execution environment.
18

---

005 Evatract tomer Dvel (Aeg. 0.345) costrace
(nog
0.244)
Probablity Density
0
0.2
0.25
0.3
0.35
0.4
0.45
0.5
0.35
0.65
Edit Similarit
Figure 5: Edit similarities between PYX and two popular benchmarks
14.32% (Domain Specific Problems)
10.63% (Mathematical and Computational Problems)
50.57% (Algorithmic and Data Structure Problems)
10.58% (Data Science and Machine Learning Problems)
4.69% (Performance Optimization Problems)
2.71% (Database and SQL Problems)
1.87% (Security and Cryptography Problems)
1.8% (User Interface and Application Design Problems)-
1.51% (Web Problems)
1,43% (System Design and Architecture Problems)
Figure 6: The category distribution of PYX
Data Statistics We perform decontamination the PyX, and monologue samples (we remove the samples that share the same input and output pair with CRUXEval). PYX includes 32,489 natural language code pairs. We generated 29,945 forward monologues and 31,022 backward monologues using rejection sampling. SEMCODER is trained with 93.4k samples, and SEMCODER-S with 214. lk samples. PyX-R contains 18.473 debugging samples, each with the original description, buggy code, debugging rationale, and final patch.
Example 1: An example of PyX sample generation
User Prompt:
folloxing code snappet create high quality programming problem. Present your output in twa distinct sections: [Problan Description] and [Solution).
gaan inspiration
Code anippet for inspiration:
testinfra_ hootss
www.saaansnseeeeeeeeeeeeeeeeeeee AnribleRunner
environ [" ("MOLECULE, INVENELRYYYOTOTY_ILEE. getth (awooastta""nsataaa ("instance")
def test_non__rof
(host):
with sudo sudo (user=' (user=" vaggrant")
container run helle world") agsert "Hallo fron Dacker!" in cnd stdout
run_tess (command-
def test wost_docker_gre www.gggrroorerrreeeeeeeeeee (bost):
hoat. (path=* group content_atring
match search(
pattern=r"" afrtemmrrr vagganttt,,,
stringscontent, flags-re. NULTILINE.
agsert match
19

---

Guidelines for each section:
[Problen Description]: This should be concise and ssconpletely self- solf-containeds4, providing all the contextual information one needs to understand and solve the problen. Assune progranning knowledge,
any specific context, variables, code snippeta pertinent to this problen are explicitly included. Avoid interaction with external resources (e.g. keyboard, uncommon third-party libraries. The problen should be non-trivial and require some thought to solve.
systen, databases, network) requirement
2. [Solution]: Pirst, show your thought process and outline your planned steps clearly. Then offer comprehensive **coreett* and the (Problen Description] you provided.
accurately addresses
Model Response:
(Problem Description] Write solution
folloving coding problen:
You are tasked with creating Python function that processes given string to extract apecific infermation based predefined patterns. fuaction should take string as imput and return dictionary containing counts of various elements found in the imput string.
The functica should periora the folloving tacks: Count occurrences 2. Identify and count the occurrances of email addresses present in the input string. Determine the total count numeric digita (0-9) in the input string.
unique
input string.
For example, given the input string "Hello world, my email is exampledemail.com and ny phone number 12345", the function should return dictionary like this:
'word_counte" ('Hello': 'verld': 'ny': 'email'; ' con': 'and": 'phone's 1, 'number's 1), 'ensil,count 1, 'digit_count':
'exampledemail.
Ensure that the function is case- inseniiseii when counting words and email addresses. sequences characters separated by spaces, follow the standard format of "exanpletemail.com"
Nords
[Solution]
To solve this problen, se will create a Python function that pracesses the input string specified requirements. Here is outline of the planned ateps: 1. Initialize variables to store word counts, email count, and digit count.
according
Tokenize
inpat string into vorda.
3. Iterate through the vords count occurrences and identify enail addresses. Count the total nunber of digits in the input string. Return dictionary containing the vord counts, email and digit count.
Python function implementing solution:
"python import re
def process_string (tapttn((tttteerrann)) cord_counts = O
amail_count
digit_count
words input_string lover appl:)
for word words:
if re. natch (r* [a-zAA2]])) word): vord_counta [sord]
if ze. ne.chtch(s (I" [a-zA-20- 20- (werd,
word);
email_count
digit_count += sun(1 for char in word if char. indigit
{
"zord_counta"" coun_co eensil_count', email_count, 'digit_couat'; digit_count
Teat the function with an example lnput input_atring "Hello vorld, anail examplademall and ny phone number 12345* result - procese_strine (input_atringg print (resutt)
20

---

This function processea imput string described prablen returns dictionary with the counte vords, email addressea, and digits. The regular expressions used help identify words, emsil addresses, and digits accurately.
Example 2: An example of PyX-R rubber-duck debugging and bug fixing
Prompt:
a** Problem
Write solution the following coding problem:
You are tasked with implamenting Python class that pracesses and represents data from HTML soup objects. The class thres nethods: "..init.."" "..str."" The "._init.. nethod takes "soup" object, remores empty elements, and parses the input -.stt.-"
"..repr..
"..repr-- nethods return object data
fornat.
Your task is to complete the implementation of the class by defining the .renovo_aapty_a.. _appty_elements and "apars__aapat methods. The -ranove_enpty_elomeee nethod should ranove any anpty slementa from the "soup" objact, and the "pparae_input nathod should process the "soup" object
relevant
Implement the missing methods to ensure that the class functions correctly outputa data in JSON format when the -._trr. -..rep_. methods called.
Solution
python
from beg inport BeautifulSoup
import json
class HtnlDataProcessor:
def __init__ (self, soup):
soup soup
self.data www._reeeeettt.eeeee.ee self innt.
def _renove_enpty enteeeetatyy.lementa elements (self): Remove anpty tags for element in if not self. solf.aoup.ffndd aoup.find_allltrue)) getatt.ge__eeee get_text (stzip-True):
the soup
# True rinda all taga
elenent is empty element. (econpose # Renove it from the soup
def _parse_ parse__ (sell(;
Extract relevant data from the soup self
for element in self.soup.. wo...oiddddee................ find,all (True): # Iterate over all tags if element nane in self data: self dalemne nane] append (element. getggetttttttteeeeeeeeeeeeetteee (strippTruue))
delf.data delr.data [=lement. name] [element, get. (atrip-Trre))
def __str._ (self):
return json. dumps (self. eueure_aecii-Fall
def (self):
return jaon dumps (self data, ansure_aacii=Falae
ava Failed Test "python
Teat cases to verify the implementation
I Test with simple HTML soup
content "<div><p<eelo<</>" http:///w....lpppppp//p/////////
soup BeautifulSoup cottl_coeet 'html parser))
processor RaalDatsProeessortenn (soup)
str (procassoo)) '("div": O, "p": ['Hello"], "span": ["Norld*]]'
Target:
21

---

[XCHOLOGUE]
#*a Execution Simulation
salk through execution provided teat case buggy HtalDataProcessor implementation.
**Ioput HTML*::
imput
"tht
Thia geta parsed into Boautifulsoup object "soup".
*eInitialize HenlDatsProcessor" *:: Nhen parsed "talDataProcessrr (soup)" is called:
"self. soup"
self. is initialized as an empty dictienary.
senovee_ontty.llemntte
-parse__nputt))
sequentially.
Execution enpty. all elements "soup":
The nethod iterates
"ddi>"; Not enpty (it has children), so it renains. enpty, ><><</>>>; Empty, so it gets decomposed (renoved). "<ppn>>orld<<<pam>""
'<p>Hallo</p>',
empty, ao remains. After this proceas, the "soup" nox looks like: html
4. ** Execution of -parse__nput **: iterates over tags again:
For "<div>"; It adda child elements texta, resulting in: "pyyhhn
self dalf.data('div"' ['Hellokor1d']
<<p>";
"Hello":
"pyyhon
self.data ['p'] ['Hello']
For "<span>": It adds its text "Norld"; ". "'python
self.dataf dataddaassssaa I'span '] [
At the and of this mathod, "aelf. data" is:
ppytoon
('div': ('BelloWorid'), 'p': ()Hello',, 'epan': ['World '])
5. *string Representationn::
"_.str.. nethod is called, producing:
jaoon '("div"; ("HelloWorid"), "p": ["Hello"), "span"; ["World"]}'
"assert (proceaoo))) == '{"div": [], "p": ['Hello "], "apan": ["Vorld"])'- fails because the expected output for "div" is an enpty list, while generated output "HelloNorld""
**Test Assertion*::
ava Root Cause Analysis
The prinary issue lies within the _parse_input method. Specifically collects text from child elenents under tags like '<div>" which makes "self data div 'I" incorractly separately.
aggregated with
its children,
displaying
Bug Pixing
To fix this, de should prevent the parent tag from aggragating text that includes child clements. Instead, se should only include text for elements that do not have child elements in "-parse_iaput" ", Here's the corrected inplementation: [/NDxaLOGUE] [Refined]
python
22

---

Daly process elements without child inport BesutifulSoup import Joon
class HtalDataProcessor: soup):
def soup soup self.data - ()
www.rrreeoee............ paat...arre.nnppt.
def eraooee.nptyyeeeeeete
Renove enpty tage from the soup self aou.. if not element olenent.gett.ext get_text tet.tetttettttteteeeeteeeee (ruz: 8 If the element is enpty soup
tage
decompose()
Ranove
def input
Extract relevant data from the soup self.data ()
for element self, soup. find_all (Trua): Iterate all tage if not element olement.find find_allddllllealaaaalllllll (lrue)
elements
element_ text element gettt..eettttttteeeeee (strip=True)
if element nane in self data:: self data (elemeennn
append (element_text)
clse:
self data [elenent. nane] [element_text]
else:
if element name children, you initializa ampty self.data: self cata []
def __str__ (self):
jaon dumps (self seeef.ddae ebsure_aeci1-aale))
def -..rpp__- (aelf):
return jeon. dumps (self, date, ebeure_aecli-Palee)
[/Refined]
G Detailed Example of Monologues
We provide examples for the detailed forward and backward monologues.
Example Monologue Reasoning
Code and Input:
typing import
def mnique_eorted...
inddddiiiiieeeeeeiieeeeeee List [float]) >> List (int);
energy
for idx, anergy in mumerata(anerriaa
serrgy.dctt.aatdrruuu. (energy, idx) sorted__nique_enee unique_sorte sortad__ndddceee [anergy (iettrrrtrtttttry for anergy in sortod__uiqqueenergtaa
sorted (energgien)))
unique_ indices assert unique_sorted_inde (100.5, 8.2, 10.5, 7.1, 8.2)) "
Forward Monologue:
23

---

[XONOLOGUE]
#** Understanding the Function
function unique_ indiccs takes energies" input. Ite purpose is to find the indices of the unique elenenta in the imput list when sorted in ascending order.
floating pecating--ointt point nombera named
dictionary "energy.dict occurrence indez,
nap unique energy
first
enumerate" function used iterate through dictionary with energy values as keys and their respective indices as values. Then, the fuactien creates sorted 1ist unique energy valuea from the input list using (ortee((et(eneergiee)) Finally, it creates list of indices that correspond to these sorted unique energies compreheneion that
and populate
using
original indices
dictionary.
Key Properties
The input list can contain duplicate values and is of variable length. The output list will have unique energy values sorted in sscending order and will contain their originsl indices from the input list.
If an energy value is repeated, only its first occurrence 's index is stored in the dictionary.
The function ensures that the indices in the output list reflect the order of the unique
after sorting.
a** Execution Simulation:
Let's simulate the execution of "mique__orted_iidiee ([10.5, 8.2, 10.5, 7.1, 6.2]) step
by step.
1. winirialiizaionn::
-'energy_dect initialized an anpty dictionary.
*fFilling the Dictionery+e:
The "for" loop begins, iterating through the list with 'onunerate's
**Ittration ies: For 'idx 0", "anergy 10.5", . The dictionary is updated to (10..5: 03"
**Iteration 2+x: For 'idx 1", "anergy 8.2". The dictionary is updated to (10.5: 0, 8.2: 13"
#4lteration Se*: For 'idx
"energy 10.5", Since 10.5 already existe in the
dictienary, it is not updated.
**Itrration 4+s: For iidx 'energy 7.1", The dictionary updated (10.5: 0, 8.2: 1, 7.1: 33"
*Iltrraioon dictionary, it is not updated.
4', "energy 8.2", Since already
Finsl state of "ieergy-diet" after the loop: (10.5: 0, 8.2: 7.1: 3)"
s*Sorting Unique Energies**:
The unique energies are derived and sorted, reaulting in [7.1, 8.2, 1
*<Retrieving Driginal Indicesss:
list comprehension retrieves the original indices corresponding these sorted
unique energies:
"energy 7.1", inder
For "energy 8.2", the index is "1"
"energy 10.5",
'0'
Thus, ceappe__orte__iddiioi becomes [3, 1, 0)*
Retuuurn Statenent**;
The function returns the list [3, 1, 0]*
[/NDXOLOGUE]
[ANSWER]
assert
8.2, 10.5, 7.1, 8.2)) - [3, 1, 0]
[/ANSVER]
Code and Output:
from typing inport List
def dasguee_crtee__ndicee (enrgeeee::: List (iloat]) >> List(int):
enorgy_dict O anargy cetted_uaiqu_eenerrgia sorted (set (energiess)
munerata idx)
unique_torted corted_indices [energy_dict [energy] for energy in sorted__uiguu_cnorggaee
return unique_soree sorted indices
meique__orted_iidiieee
[3, 1, 0]
Backward Monologue:

---

Underetandiag
indices constructed by retrieving original indices from "energy _dict" correspending to the sorted unique energies. indices corresponding to unigue The function "nique_ cortad_indicer processes list of floating point numbers (' anergies") produce
unique
ascending order. Here's
hos it works:
1. **Initielization** An empty dictionary "energy_dict" is created.
function iteratea through "energies" using "enunerate", which provides both the index ( ("idx') ) and the value ("energy") uses "setdefault" to list. If the anergy value already exists in the dictionary, it does not update the index (thua only the first index stored).
saDictionary Population<s:
each unique energy
3. #4Borting Vnique Energieg*s:
of unique energy values is created from the list, _eerrgeeessss" "iree_uuiiiueeoorriie""""""
which is then sorted to create "orted__nique__nn
Valuete: function returns anergy values in sorted ordar.
ava Key Properties and Constrainte
The input list may contain duplicate energy values, but only the first occurrence's
index will recorded in "energy_dict"
The output will be ordered according to the sorted unique energy values. output
length input 1ist.
will equal number unique energy
ата Reverse Deduction
Given the output [3, 1, 0]', ea need to determine what ennrrgass might have been to produce this result:
**Behind the Resultes:
-enerryy
muat map some unique energiea
indices.
The first unique energy corresponds to Index "3", the second unique to "1" and the third unique '0",
**Energy Values+::
Suppose the values in the "nergies'' list that correspand to these indices are: "energiea [3] first unique energy (let's call it 'el')
-eenrrgees [1] -> second unique energy (lat's call it 'e2")
"energiia [0] third unique energy (let's call it 'a3")
**Expected Orderinges:
In the sorted list of unique energies, "el" must be the smallest, folloved by 'e2",
and then 'e3"
x*Choics of Valuesex:
siuple
array can created.
The energy at inder "3" in less than the energy at index "1" and both should be lesa than the energy at index "0"
Let's select:
"enerrgien [3] 1.0" (first unique, snallest value)
"energiea [1] 3.0" (second unique, niddle value)
"energiee 5.0" (third unique, largest value)
To ensure proper indexing and repetition, ve can fill the rest of the 1ist with duplicates such values, for instance:
... python
anergias [5.0,
1.0]
index mapping: "6.0" at index "0" (first occurrence),
arrengenent
"3.0" at indez '1' (first occurrence),
"1.0" inder "3" (first occurrence).
[/NDNOLOGUE]
[ANSWER]
unique ((5.0, 3.0, 1.0)) [3. 1,
[/ANSVER]
H Task-specific Prefix
We append task-specific prefixes the training samples to guide the model to perform different types of semantic reasoning.
Task: Natural Language to Code
Prefix:
25

---

You are exceptionally intelligent coding assistant that consistently delivers accurate reliable according <tl_Descriptton>
Deccripottt (proupt)
<Code> (completion)
Task: Forward Monologue
Prefix:
Simulate the Execution: You are given Python function and an assertion containing function input. Complete the assertion containing the execution output corresponding to the given input in [ANSHER] and [/ANSWER] tags. (proupt) [completion]
Task: Forward Monologue
Prefix:
Deduce Sanantic Constrainta: You given Python progran expacted output. Find one input such that executing the progran with the input leada to the given output. Complete assertion {prompt) (completion)
input [ANSHER] [/ANSWER].
Task: Dehug Self-refine
Prefix:
"*"Dabug and Refine the Coda: You are given problem to be implemented in Python, and buggy code that tries to solve the problen but faila test caae.
You should firstly simulate the execution with the buggy code and the failed test identify the root cause of the failure. you should fix bug wrap refined code
[Refined] (/Refined
].
(instruction)
(response)
I Baseline Trace Formats
We present the baseline traces formats as we discussed and compared in Section 6.2
Source Code and Test Case
from typing Inport List
[12]
def unique_sorten indices (anergies: [ist(fl >> List [int]: [L6]
energy_dict
[16]]
for idx, energy in (uunerate(energiee) [L7]
(energy, idx) [LB]
unique _neggieeeeeeeee corted (energeles))) [19]
unique_eorte__indice. [energy_diet [eneryy) for energy in sortee_.uiquu_ennrriie: [L10]
raturn unique_sorted_ indices [L11]
assert unique_sorted. www.eoteeddddddeeeeeeeeeeeeeeeee indices ([10.5, 8.2, 10.5, 7.1, 8.2)) - [3, 1, 0] [L13]
Scratchpad
26

---

typing import
unique indices [iat(float]] [int]: [IMPUT] ("anergiaa": [10.5, 8.2, 10.5, 7.1, 8.21) [/INPUT]
energy_dice O [STATE] ("energy_dict"" () (/STATE)
energy [STATE] ("idx": 1, "energy": 8.2) (/STATE] (STATE] ('idz*; 2, "energy". 10.5) [/STATE] STATE] ("idx";
(energiee::: [STATE] ("idx"; "energy"; 10.5) (/STATE
"energy"s 7.1) (STATE) ("idx"; "energy": 8.2) [/STATE]
energy. www.g.ddct.ssseettaee setdefault (enerr (energy idx) # [STATE] ("energy_die) dictt: *[10.5: 03") t/
STaTe] [STATE] {"energy senergyddict". *(10.5: 0, 8.2: 1)") [/STATE] [STATE] ("onorgy-dies";
"(10.5: 0, 8.2: 1, 7,1: 3}"} [/STATE]
sorted (st(erggiee))
[STATE) wnw.rtd__iiiuooeeeerggee
[7.1, 8.2, 10.5]) [/STATE]
[STATe] taaiqueeortte_iidiceeo return ctt_ssoorrdd_ddceee [OUTPUT] [3, 1, 0] [/auTeUT]
lenergy.dict (energy] for energy in sorted sarted_uniquo_enerria. "ndiccee [3, 013 [/STATE]
NeXT Scratchpad
from typing inport List
def unique_ sorted indices
List [float]] <> Liat[int]] [INPUT] ("energies";
[10.5, 8.2, 10.5, 7.1, 8.2]) [/INPUT]
enorgy_dict O # [STATE-O) d'energy_dict": () [/STATE-0]
idx, anargy enumerata (onargiee)) [STATE-1] {"idx": 0, "anargy": 10.53 I/ STATE-1) [STATE-3] ("idx": 1, "energy"; 8.2) [/STATE-3]
[STATE-8] ("idx": 4, "energy":
8.2) [/STATE-8]
soorg_ddact.eetdffare (energy. . idx) [STATE-2 -2] ("energy_dictt" "(10.5: 03") I/ ("energy_dict
STATE-2] [STATE -4] ("energy_ diet": *(10.5: 5: 0, 8.2: 1) ((0055:
1,
3)*} [/STATE-1
corted_unique _nerggees (set (seeeegggaa))) # [STATE-9] ("aorted_uniiuu_cnnees
[7.1, 8.2, 10.51) [/STATE-9]
unique_eorted__ _orteed indices
diet for
in unique_ *
(STATE 10] ("unique_ ported_indiced [3, 1, ol) (/STATE
unique_ indicas [OUTPUT] [3, 0] [/OUTPUT]
Concise Trace
[16] [INPUT] ("enargies"; [10.6, 8.2, 10.5, 7.1, 8.21) [/INPUT] [/L6]
[L6]
""energy-dict""" {}} [/L6]
[17] {"idx": 0, "energy"; 10.5) [/L7]
[18] ("energy-diets
"(10.5: 0}"} [/18]
[17] ("idx": 1, "energy"; 8.2) [/t7]
Denergy_dict'? "(10.5: 8.2: 1)") [/18]
[17] ("idx": 2, "energy": 10.5) [/L7]
[18] [/L8]
[L7] {"idx": 3, "energy": 7.17 [/L7]
[18] ("energy_diets
"(10.5: 0, 8.2: 1, 7.1: 3)*) [/18]
[17] ("idx": "energy"; 8.2) [/17]
[18] [/L8]
[/L7]
[19] ("sorted_ unddduniquuu..e _niqueeenargiess. [7.1, 8.2, 10.5]} [/19]
[110] {"unique_ cornee_sorrdd__iee sortad_indicas"
ol} [/L10]
[L11] [OUTPUT] [3, 1, 0] [/OUTPUT] [/L11]
27

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors their evaluation. While "[Yes] " is generally preferable to "[No]" it is perfectly acceptable to answer "[No]" " provided proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often
nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The summarization of the paper's contribution and novelty is in Abstract and Introduction (Section I
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
28

---

Justification:
Appendix A
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sellispecifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: There no theoretical results in this work since results focus on empiri- cally improving and evaluating Code LLMs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have explained the configuration of our implementation and configuration. We will also release the reproducible artifact publicly.
Guidelines:
29

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our data, code, and models are available at: https://githut. com/ PorpsssooooooPPPP/eeeeeeee
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nis.cc/public//uust http://///giiiee//eeeuuubiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
30

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We include data collection and experimental configuration in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We fine-tune 6.7B transformer decoder model and use greedy decoding for our main results in Table There's 's no need to report statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: In Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
31

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///aaaaaasscccpp/////ccccca
Answer: [Yes]
Justification:
We checked the code of ethics and ensure our research algins with the code.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix B
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Appendix B
32

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Our data is synthesized by GPT-3.5, and have explicitly mentioned that in the paper, and all related works we borrow ideas from are cited
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pacerswithcode thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Our documents will be available at https://github. com Cotpsttiubbccm//RR........ AAE/ERRREEEEEEEE LARiSE-Lab/
SemCoder
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
33

---

Answer:
[NA].
Justification:
There's no crowdsourcing nor human subjects in our paper.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA].
Justification: There's no crowdsourcing nor human subjects in our paper.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "8m7gpQgWfr", "reviewer": "Reviewer_GuSF", "summary": "The paper proposes PyX, a training dataset focusing on preserving the semantics of sample programs, and SemCoder, a coding language model trained with PyX with better ability to capture program semantics. The construction of PyX dataset comprises three phases to ensure the code samples involved are correct and self-contained. The dataset also contains pairs of buggy code, faulty traces and patches to improve the ability of LLMs in terms of debugging and self-refining. Based on the PyX dataset, a coding LM SemCoder is trained with a technique called Monologue Reasoning to improve the ability of code reasoning and comprehension in forward and backward directions. Comprehensive experiments are conducted to evaluate the performance of SemCoder, showing the SOTA performance and self-refining ability of SemCoder, along with the effectiveness of the training techniques involved.", "questions": "+ During the construction of PyX, sample programs are filtered following certain criteria (e.g. Python built-in types only). However, user-defined types such as classes are crucial and frequently-used in real-world programs. Will this act impair the generalizability and applications of PyX and SemCoder? How can this gap be tackled if possible?\n+ During the construction of PyX, a generator model is adopted to generate pairs of problems and solutions. Is there any potential bias that the model tends to generate outputs with certain limitations (e.g. low complexity, or high similarity to the training dataset of the model)? If so, to what extend does it eventually impair the generalizability of PyX? \n+ It seems that the monologue reasoning technique (especially the forward one) is targeted at concrete inputs and outputs of programs. What about abstract inputs and outputs that conforms to certain specifications? Can the technique improve the summarization (of abstract inputs/outputs and specifications) ability of LLMs , and how? Such ability can be crucial under certain tasks such as documentation/specification generation.\n+ How does monologue reasoning improve the code comprehension ability of LLMs? Does the effect of monologue reasoning rely on the explicit textual analysis in the outputs of LLM? More specifically, how does the improved code reasoning ability of SemCoder comes into play under common coding tasks such as code completion, especially when explicit textual analysis on input code is not allowed? I expect to see more analysis and intuitions on this regard.\n\n+ Can the author provide more examples and intuitions on the debugging and self-refining process of SemCoder? What kind of information is given to the LLM as feedback? How does the LLM utilizes the information? To what extend can the LLM identify the root cause? How effective is the LLM in terms of fixing the bugs?\n+ Also, can the debugging ability of SemCoder be adopted into tasks like program repair? Is it possible to conduct more thorough evaluations on this regard with benchmarks more targeted on APR?", "limitations": "See weakness", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ A novel and open-source dataset to tackle a widely concerned problem\n+ A language model trained on the dataset with SOTA performance\n+ Easy-to-understand and effective ideas and techniques for the training process\n+ Clear presentation", "weaknesses": "+ Hard to explain how monologue reasoning improves the performance of common coding tasks\n+ Limited intuition on the experimental results, e.g. how does SemCoder conduct self-refining", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed reviews and questions! \n\n## Self-contained Programs with Built-in Types\n\nThe primary goal of SemCoder is to propose a recipe for learning comprehensive semantics of code with LLMs. To achieve this, we initially focused on collecting self-contained code snippets with built-in types, allowing for a controlled environment to effectively extract semantic signals. This design choice allows us to collect code as well as generate high-coverage unit tests,  achieving thorough execution and efficient tracing to expose comprehensive semantics.\n\nHowever, SemCoder is a general approach and can be extended to code snippets with additional complexity. Existing context retrieval tools and program analysis techniques, such as program slicing, can be utilized to retrieve the necessary out-of-scope context, such as cross-file dependencies and user-defined types. When a complete context is provided to the model, the monologue reasoning is generally applicable, regardless of whether the code contains only built-in types or user-defined types. On the other hand, we acknowledge the difficulty of generating thorough test cases for complex programs, which remains an open challenge in software engineering. Therefore, we regard such augmentation of PyX dataset as a future work.\n\n## Bias in Problem-Solution Generation\n\nThe potential bias introduced by the generator model for creating problem-solution pairs is possible when freely prompting the LLM. We mitigate this bias by providing diverse seed code snippets as the prompt for LLM to generate diverse response autoregressively, inspiring the a varied sampling process. As a result, PyX samples cover distinct topics and scopes. We show the category distribution in Figure-5 of Appendix, where each category could contain hundreds of distinct topics. We will add a diversity discussion with more plots and details in the next revision.\n\n## Extension to Abstract Input and Output reasoning\n\nCurrently, SemCoder was trained with code samples accompanied with concrete inputs and outputs for two main reasons: (1) we target to expose the dynamic program states with real execution for code semantic learning, which requires concrete executable inputs (2) we could validate the correctness of monologue reasoning (though not perfectly) by comparing its concluded input/output with the real execution results. \n\nHowever, as we introduced in Section 4.2, the monologue is constructed with the flexibility of including both operational and abstract semantics, where the latter describes abstract specifications or properties. Though such abstractions are more dominant in the backward monologue than the forward monologue in PyX, it should be straightforward to strengthen such abstractions by constructing a training dataset with abstract specifications as input and output and fine-tune SemCoder to drive its focus towards abstract semantics.\n\n## The Effectiveness of Learning Monologue Reasoning\n\nWe would clarify that the LLM’s general code comprehension capability gets improved with monologue reasoning training, regardless of the presence of explicit textual analysis during the inference. There are two main pieces of empirical evidence. \n\nFirst, in Table-1, SemCoder achieves state-of-the-art performance in code generation among <15B code LMs, significantly outperforming our base model DeepSeekCoder-6.7B-base. The code generation evaluation does not explicitly perform monologue, so this improvement illustrates the monologue reasoning produces a more capable programming system as it not only learns to generate code but comprehensively understands its semantics. However, as we have discussed in the \"Appendix A - Limitations and Future Work\", we acknowledge that the explicit monologue might further improve the accuracy of code generation.\n\nSecond, as illustrated in the caption of Table-1, we evaluate SemCoder on CRUXEval with two settings: direct prediction (without monologue reasoning) of input and output (left side of “/”) and prediction with monologue reasoning (right side of “/”). We notice that SemCoder achieves notably better performance even without explicit monologue, highlighting the overall improvement of execution reasoning capabilities.\n\n(Due to the character restriction, we answer the last two questions in the following comment)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zMo0jP7mn7", "reviewer": "Reviewer_sURP", "summary": "This paper addresses the limitations of Code Large Language Models in understanding deeper semantics such as execution effects and dynamic states. It introduces a novel training strategy to enhance Code LLMs with comprehensive semantics, including high-level functional descriptions, local execution effects, and overall input/output behavior. Using a clean code corpus called PYX, the approach trains Code LLMs to write code and reason about execution behaviors using natural language, mimicking human verbal debugging. The results demonstrate the effectiveness of integrating semantics from multiple dimensions and improving debugging and self-refining capabilities.", "questions": "Please check the questions in the weaknesses part.", "limitations": "N.A.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The idea of introducing different semantics to augment code LLMs is quite interesting.\n+ Using various kinds of semantic information to enhance code LLMs is a commendable approach.\n+ The experiments effectively demonstrate the model's capabilities across different aspects.", "weaknesses": "- Some descriptions lack clarity.\n- There are no use cases demonstrating the model's usefulness.\n\n1. In section 6.2, how did you fine-tune the base models? What are the data generated for NeXT and Scratchpad, including both forward and backward monologues? Line 331 only mentions the backward monologues.\n2. In Table 2, what metrics are used for evaluating input prediction, especially how can you evaluate with your abstract input representations? Can you show some trace examples of backward prediction for different baselines in the table? It is not easy to understand why using abstract semantics works better. Some case studies would be helpful.\n3. In Table 3, why was NeXT not selected as a baseline, given its focus on repair? What are the differences between CodeLlama with PYX-R and SEMCoder (-S)? Since the base models are different, why wasn’t SEMCODER fine-tuned on CodeLlama and Magicoder for fair comparisons?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s detailed comments and feedback!\n\n## Baselines in Section 6.2\n\nWe apologize for the confusion. We provide an example of baseline trace formats in Appendix H. Given that neither scratchpad nor NeXT provides replication packages, we re-implement the approach following their illustration in the paper. \n\n**Data Format**: For both scratchpad and NeXT, we trace the execution and construct a dictionary for the changed variables after each execution step. We wrap such a dictionary in between two special tokens `[STATE]` and `[/STATE]` to represent the change of the program state, and if a line is executed multiple times, we will concatenate these states by the execution order. Finally, we append execution states to corresponding code lines as comments, as suggested by NeXT. The differences between Scratchpad and NeXT are two: (1) NeXT improves Scratchpad by explicitly indicating the sequential order of program states with numbers (2) NeXT ignores the intermediate states within a loop and replace them with `...`\n\n**Fine-tuning for Input and Output Prediction**: For baselines, we finetune the same base LLM (deepseekcoder-6.7b-base) as SemCoder to predict input and output but with the intermediate execution traces using the corresponding format shown in Appendix H. \n\nThe model input of all baselines is the same, as shown in the “Source Code and Test Case” section of Appendix H, containing the source code and one executable input. The target sequence for the output prediction will be the inlined program states as shown in “Scratchpad”, “NeXT Scratchpad”, and “Concise Trace” sections. For the input prediction, we will reverse the line order of the inlined program states as the target sequence: for example, the scratchpad for input prediction will be:\n\n```\n    return unique_sorted_indices # [ OUTPUT ] [3, 1, 0] [/ OUTPUT ]\n    unique_sorted_indices = [ energy_dict [ energy ] for energy in sorted_unique_energies ] #\n[ STATE ] {\" unique_sorted_indices \": [3, 1, 0]} [/ STATE ]\n    sorted_unique_energies = sorted(set(energies)) # [STATE] {\"sorted_unique_energies\": [7.1, 8.2, 10.5]} [/STATE]\n    …\ndef unique_sorted_indices ( energies : List [ float ]) -> List [ int ]: # [ INPUT ] {\" energies \":\n[10.5 , 8.2 , 10.5 , 7.1 , 8.2]} [/ INPUT ]\n```\n\n## Advantages of Abstract Semantics\n\nThere are two main advantages of abstract semantics by design compared to scratchpad-like execution traces. \n\nFirst, it properly handles the uncertainty in the program execution. For example, in the above example of backward execution reasoning, the model needs to deduce `energies` from its sorted set `sorted_unique_energies`. The scratchpad-like trace provides only one example of `energies` while the candidates could be multiple. In contrast, the backward monologue describes the high-level property of `sorted_unique_energies`: “It should be a disordered list with at least one 10.5, one 8.2, and one 7.1”, which teaches the model about the side effect of `sorted(set(energies))` rather than simply predicting one candidate. Second, abstract semantics maintain the flexibility of reasoning about key properties of dynamic program states rather than redundantly predicting details. For example, when reasoning about a tensor operation, such as transpose, the abstract semantics will focus on the important attributes like the shape, while not replicating numeric values.\n\n## CRUXEval-I Evaluation\n\nAs we have introduced in Section 4.2.2 (line 226-230), though the backward monologue will reason about abstract semantics, we force the model to conclude one __concrete executable input__ that satisfies the constraints deduced from the backward monologue. Therefore, we could validate the correctness of the prediction by executing it and compare with the expected output, similar to the standard setting implemented by CRUXEval-I.\n\n## NeXT as Baseline\n\nUnfortunately, NeXT does not provide public artifacts. NeXT is a complicated system, and reproducing it from scratch requires at least (1) their training data: 10,047 program repairs from MBPP-R (2) the pre-trained weights that initialize NExT: Palm-2-L (3) the implementation of their self-training loops. However, none of these are publicly released; without the data and the initial checkpoint, it is nearly impossible to reproduce the model’s capabilities for a direct comparison in our setting. In addition, their evaluation data: the 1,468 held-out repair tasks from MBPP-R and the full set of HeFix+ are not released, either, so we could not evaluate SemCoder using the same evaluation setting as theirs.\n\nThe most doable comparison is to compare their “naturalized execution trace” with our proposed monologue reasoning, so we refer to NeXT figure-2 in the original paper to implement the inlined execution trace as a baseline in Table-2.\n\n### SemCoder with CodeLlama\n\nThanks for the suggestion! Due to the limited computing resource, we construct SemCoder only with DeepSeekCoder as the proof-of-concept. However, the approach is generic and model-agnostic. To illustrate such generalizability and perform a fair comparison with CodeLlama, we train SemCoder-CL using Codellama-python-7b as the base model, and below is the result:\n\n| Model | CodeGen-HEval(+) | CodeGen-MBPP(+) | CRUXEval-I | CRUXEval-O | SelfRef-HEval(+) | SelfRef-MBPP(+)|\n|----------|----------|----------|----------|----------|----------|----------|\n| CodeLlama | 37.8 (35.4) | 59.5 (46.8) | 37.3 / 40.4 | 34.6 / 34.0 | 66.9 (56.1) | 72.6 (68.3) |\n| SemCoder-CL | **73.2 (67.1)** | **69.0 (57.1)** | **43.4 / 49.9** | **38.2 / 51.1** | **82.3 (79.3)** | **78.3 (78.0)** |\n\n`CodeGen` is Code Generation performance in Table-1 and `SelfRef` is the Self-Refine performance in Table-3.\n\nWe can see that the SemCoder method significantly improves CodeLlama’s capabilities in code generation, execution reasoning, and debugging&self-refine, illustrating the effectiveness of SemCoder.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Dx3zfZGxB", "reviewer": "Reviewer_MYsU", "summary": "Recent advancements in Code Large Language Models have primarily focused on code completion, yet these models often lack the ability to comprehend deeper semantics, such as the execution effects and dynamic states of code. This paper presents a pioneering approach aimed at enhancing Code LLMs' understanding of complex semantic tasks such as debugging and program repair. The methodology integrates high-level functional descriptions, local execution effects of individual statements, and the overall input/output behavior into the training of Code LLMs. This integration bridges the gap between static code text and dynamic execution states. The authors introduce PYX, a curated code corpus of fully executable samples with functional descriptions and execution tracing. They train Code LLMs to simulate human-like debugging by writing code and articulating execution behaviors in natural language. This approach culminates in the creation of SEMCODER, a specialized Code LLM with 6.7 billion parameters. SEMCODER demonstrates competitive performance against GPT-3.5-turbo in code generation and execution reasoning tasks, achieving 81.1% on HumanEval (compared to GPT-3.5-turbo's 76.8%) and 54.5% on CRUXEval-I (surpassing GPT-3.5-turbo's 50.3%). Furthermore, the authors explore the effectiveness of SEMCODER’s monologue-style execution reasoning versus concrete scratchpad reasoning, revealing a smoother integration of semantics across multiple dimensions. The study also underscores the potential of leveraging learned semantics to advance the debugging and self-refining capabilities of Code LLMs.", "questions": "This paper introduces a novel strategy to train Code Large Language Models (Code LLMs) that significantly enhances their understanding of complex tasks such as debugging and program repair through the integration of high-level functional descriptions and execution effects. The authors have developed SEMCODER, a model that performs competitively with GPT-3.5-turbo in standard code generation. \n\n- Limitation to Code Generation:\n\nWhile SEMCODER excels at tasks related to code generation and execution reasoning, its application to other areas of software development, such as code comment generation, remains unexplored. I believe the code semantics may be helpful in these areas.\n\nOverall, the idea of this paper is good. I advocate to accept this paper.", "limitations": "Please see Questions, thanks.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "+ Important area.\n+ The authors integrate the code semantics into SEMCODER.\n+ The performance is good.", "weaknesses": "- Limited to Code Generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your supportive feedback! We are glad that you like our idea of training code LMs with comprehensive semantics. Here, we would like to clarify the scope of SemCoder discussed in the paper and its potential for more software engineering applications.\n\n## SemCoder’s Scope\n\nSemCoder extends its capabilities beyond basic code generation. Specifically, it has demonstrated substantial proficiency in execution reasoning through static input and output prediction (Execution Reasoning Columns in Table-1), as well as debugging and iterative refinement (Table-2). SemCoder outperforms all baseline models with up to 2x more parameters and also approaches the performance of much larger models like GPT-3.5 in tasks other than code generation. This highlights the model's effectiveness in both generating syntactically correct code and in deeply understanding code semantics and execution effects\n\n## SemCoder’s Potentials\n\nThough SemCoder is mainly evaluated by well-established benchmarks, there are several practical applications in software development where its capabilities can be highly beneficial.\n\n**Bug Localization with Forward Monologue**: SemCoder's forward monologue reasoning could be instrumental in static bug localization. By simulating the real execution of a program line by line and reasoning about the effects of each statement, SemCoder can identify where the actual program states diverge from the expected behavior described in the functional specifications. This approach mirrors the rubber-duck debugging practice, where verbalizing the problem helps in pinpointing the issue.\n\n**Input Prediction with Backward Monologue**: The input prediction capabilities are useful for generating test cases or seed inputs for mutation-based testing. Given a desired program state to reach, SemCoder can perform backward monologue reasoning to synthesize appropriate inputs that will lead to that state. This ability is critical for ensuring comprehensive test coverage and for creating effective test cases that target specific parts of the code.\n\n**Code Comment Generation and Documentation**: We appreciate the reviewer’s suggestion of this task! Although not explicitly evaluated as a task in our paper, the monologue reasoning itself is actually a line-level code comment with a focus on the execution effect: for example, in figure-1, the forward monologues regarding line-3&4 are accurate code comments. Since SemCoder is trained to reason about both high-level functional descriptions and detailed execution semantics, we believe it could be easily extended to generate insightful code comments and documentation when fine-tuned with more task-specific data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZoJ1uHK0Gl", "reviewer": "Reviewer_H1xj", "summary": "This paper proposes SemCoder, a code LLM with code semantic-aware pre-training. The authors first use OSS-instruct to create a synthetic dataset PYX for model training, and a PYX-R dataset for training LLMs for debugging and repair. Then the authors train the SemCoder model with NL to code, forward monologue, and backward monologue tasks. Results on code generation datasets show that SemCoder outperforms a group of open-source LLMs and GPT-3.5 with fewer parameters. SemCoder also demonstrates better ability in execution reasoning.", "questions": "- When building the PYX dataset, how did the authors choose the seed snippets?\n- Why do the authors choose DeepSeekCoder-6.7b as their backend model (instead of other code LLMs or its instruction-tuned version)?\n- Can the authors discuss why SemCoder-S significantly outperforms SemCoder on HumanEval, but achieves similar results on the rest tasks?", "limitations": "The limitations are well-discussed. However, I'm still curious whether the proposed approach can be extended to real-world programs with more complicated dependencies and data types?", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "+The authors successfully trained a code LLM, and proposed novel approaches for both training data collection and training task design.\n\n+The forward and backward monologue is interesting and proven helpful.", "weaknesses": "-In Section 2, I cannot find the examples for operational and abstract semantics in Figure 1 (“the middle yellow box” and “the extreme right box”).\n\n-The approach of collecting traces seemed limited to simple programs without external dependencies and types. I wonder if SemCoder can still benefit code generation/understanding tasks in real-world data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful comments!\n\n## Seed Snippets\n\nPyX only uses parsable code snippets as seeds (Section 3.1). This is a small improvement over OSS-Instruct which randomly samples consecutive lines from code files, based on the observation that parsable seeds are more likely to yield syntactically correct programs. To do this, we first parse existing programs into ASTs and then randomly select sub-trees and use the corresponding source code as seed. As a result, we see a small performance boost over OSS-Instruct in Appendix Table-6.\n\n## SemCoder-S’s Performance\n\nWe studied samples in Evol-Instruct, which we combined to train the SemCoder-S: it notably increases the diversity of instructions for code generation, incorporating a broader scope and customized requests. Intuitively, this enhances the model's instruction-following capabilities, benefiting the HumanEval performance. However, the dominance of math problems in MBPP makes it less beneficial by only following instructions.\n\n\nAlso, Evol-Instruct includes very limited execution-related signals, such as execution results or traces. As a result, the resource to learn dynamic semantics of execution will be mostly from PyX, which is shared by both SemCoder and SemCoder-S to learn to reason about execution. Consequently, these two variants have comparable capabilities in the execution reasoning tasks. \n\n## Backbone Model\n\nWe choose non-instruction-tuned checkpoints as the base model to compare with instruction-tuning methods to illustrate the importance of learning code semantics. Among all ~7B base Code LLMs, where our computing resources could efficiently support the developing cost, DeepSeekCoder-6.7B-base was one of the most capable open-source checkpoints when we worked on this project. However, the approach introduced by SemCoder is model-agnostic, we re-train SemCoder with two more base models (CodeLlama-7b-Python & DeepseekCoder-V2-Lite-Base) to show that the programming and execution reasoning capabilities of different checkpoints can all be significantly improved by SemCoder’s proposed method.\n\n| Model | HEval(+) | MBPP(+) | CRUXEval-I | CRUXEval-O | \n|--|--|--|--|--|\n| CodeLlama-7b | 37.8 (35.4) | 59.5 (46.8) | 37.3/40.4 | 34.6/34.0 |\n| SemCoder-CL | **73.2 (67.1)** | **69.0 (57.1)** | **43.4/49.9** | **38.2/51.1** |\n| DS-V2-Lite | 49.4 (42.7) | 78.3 (65.1) |  39.4/51.4 | 44.1/46.4 |\n| SemCoder-DS-V2-Lite | **81.1 (74.4)** |**84.4 (71.4)** |  **49.6/51.7** | **47.8/60.1** |\n\n### Operational and Abstract Semantics\n\nDue to space limitations, we could not show the complete forward monologue of the motivating example in Figure 1, but we copy it below for illustration.\n\n```\n[L3]: `energy_dict` is initialized as an empty dictionary.\n[L4]: The `for` loop iterates over the `energies` list with their indices using `enumerate`.\nFirst Iteration (idx=0, energy=10.5):\n   - [L5]: `energy_dict.setdefault(10.5, 0)` stores the key-value pair `(10.5, 0)` in `energy_dict`.\nSecond Iteration (idx=1, energy=8.2):\n   - [L5]: `energy_dict.setdefault(8.2, 1)` stores the key-value pair `(8.2, 1)` in `energy_dict`.\nThird Iteration (idx=2, energy=10.5):\n   - [L5]: Since `10.5` already exists in `energy_dict`, its index is not updated.\nFourth Iteration (idx=3, energy=7.1):\n   - [L5]: `energy_dict.setdefault(7.1, 3)` stores the key-value pair `(7.1, 3)` in `energy_dict`.\nFifth Iteration (idx=4, energy=8.2):\n   - [L5]: Since `8.2` already exists in `energy_dict`, its index is not updated.\n[L6]: `sorted(set(energies))` creates a sorted list of unique energies: `[7.1, 8.2, 10.5]`.\n[L7]: List comprehension `[energy_dict[energy] for energy in sorted_unique_energies]` retrieves the original indices corresponding to the sorted unique energies: `[3, 1, 0]`.\n[L8]: The function returns `[3, 1, 0]`.\n```\n\nAs we defined in Section 2, the operational semantics refer to the individual steps in the code execution. From the complete forward monologue, we could see the execution side effect for each step, including the multiple executions of the same line (e.g., [L5] in the loop), will be reasoned by SemCoder during training.\n\nThe abstract semantics refer to the  high-level description of program behavior without concrete details. In Figure 1 “Bottom-up Monologue”, the model “deduces” the input step-by-step. Since it is in the reverse order of the natural execution, the deduction requires abstract reasoning, such as “The energy values are supposed to be stored in the dictionary, but nothing is stored”. This is in contrast to the operational semantics, where certain variables should be reasoned with concrete values (e.g., `energies is [7.1, 8.2, 10.5]`)\n\n\n### SemCoder’s Extensibility\n\nThe primary goal of SemCoder is to propose a recipe for learning comprehensive semantics of code with LLMs. To achieve this, we initially focused on collecting self-contained code snippets with built-in types, allowing for a controlled environment to effectively extract semantic signals. This design choice allows us to collect code as well as generate high-coverage unit tests,  achieving thorough execution and efficient tracing to expose comprehensive semantics.\n\nConceptually, SemCoder is extensible to real-world complexity by scaling up the computing to consume retrieved contexts and dependencies. First, monologue reasoning is designed to be general, similar to the inner monologue process when humans think. Monologues can be constructed for many manual analyses over realistic software (e.g., repository-level rubber-duck debugging). Second, the monologue is more efficient and flexible at representing complicated types and structures, compared to Scratchpad or NeXT. For example, when a variable is a tensor or very complicated class, Scratchpad or NeXT has to represent the lengthy and redundant details while monologue could simply describe the attributes that directly contribute to the downstream task, such as the shape of the tensor rather than all numeric values in it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work introduces a dataset of executable programs with execution traces and uses this dataset to train SemCoder, a 6.7B parameter code-specific LLM. Reviewers generally recognized the substantial contribution, strong performance, and various specific technical contributions such as the forward & backward monologue objective. The main weakness of note is the relatively narrow focus on code generation tasks (in contrast to other code-related task), which is in part due to the reliance on relatively simple, executable programs. This concern did not outweigh the positives noted earlier, leading to the decision to accept the paper.
Decision: Accept (poster)