What Makes Large Language Models Reason in (Multi-Turn) Code Generation?
Authors: Kunhao Zheng
Juliette Decugis
Jonas Gehring
Taco Cohen
benjamin negrevergne
Gabriel Synnaeve
Venue: ICLR 2025 Poster (2025)
Keywords: Large language Models
Multi-turn Code Generation
Chain-of-Thought
Primary area: foundation or frontier models, including LLMs
TL;DR: We break down the main components of recent CoT and self-repair techniques for multi-turn code generation identifying the most promising ones across benchmarks and models.

Abstract: Prompting techniques such as chain-of-thought have established themselves as a popular vehicle for improving the outputs of large language models (LLMs). For code generation, however, their exact mechanics and efficacy are under-explored using unified metrics and benchmarks. We thus investigate the effects of a wide range of prompting strategies with a focus on automatic re-prompting over multiple turns and computational requirements. After systematically decomposing reasoning, instruction, and execution feedback prompts, we conduct an extensive grid search on the competitive programming benchmarks CodeContests and TACO for multiple LLM families and sizes (Llama 3.0 and 3.1, 8B, 70B, 405B, and GPT-4o). Our study reveals strategies that consistently improve performance across all models with small and large sampling budgets. We then show how finetuning with such an optimal configuration allows models to internalize the induced reasoning process and obtain improvements in performance and scalability for multi-turn code generation.

Published as a conference paper at ICLR 2025
WHAT MAKES LARGE LANGUAGE MODELS REASON IN (MULTI-TURN) CODE GENERATION?
Kunhao Zheng!.2 1.2 Juliette Decugis'; Jonas Gehring', Taco Cohen', Benjamin Negrevergne", Gabriel Synnaeve' 'Meta AI (FAIR), Paris Dauphine University PSL {kunhao, jdecugis, jab)@meta.com
ABSTRACT
Prompting techniques such as chain-of ofttuughtthhht have established themselves as a popular vehicle for improving the outputs of large language models (LLMs). For code generation, however, their exact mechanics and efficacy are under-explored. We thus investigate the effects of a wide range of prompting strategies with focus on automatic re- re-prmppting over multiple turns and computational requirements. After systematically decomposing reasoning, instruction, and execution feedback prompts, we conduct an extensive grid search on the competitive programming benchmarks CodeContests and TACO for multiple LLM families and sizes (Llama 3.0 and 3.1, 8B, 70B, 405B, and GPT-4o). Our study reveals strategies that consistently improve performance across all models with small and large sampling budgets. We then show how finetuning with such optimal configuration allows models to internalize the induced reasoning process and obtain improvements in performance and scalability for multi-turn code generation.
INTRODUCTION
The field of automatic code generation has made significant progress, particularly with the devel- opment of specialized Large Language Models (LLMs) (Chen et al.] 2021 Li et al.) 2022 Rozière al. 2024 OpenAl 2023 Al Meta 2024). While these models have demonstrated proficiency in generating simple functions across various programming languages, there is still considerable room for improvement in their ability to tackle more complex algorithmic reasoning tasks, such as those found in competitive programming benchmarks like CodeContests (Li et al.) 2022). Current state-of-the-arr approaches either rely on model ensembling and massive single-turn sampling (Alpha- Code Team| 2023) or employ complex structured prompt chains for planning, editing and debugging (Ridnik et al. 2024, Islam et al. 2024). In contrast, multi-turn code generation strikes a balance between single turn approaches and prompt chains, where code is built upon previous outputs in dialog-like structure. This approach motivated by applications such as LLM-basee agents (Yao) let al. 2023b), where models tasked with decision-making and interacting with environments. In code generation, multi-turn approaches have primarily been explored on simple benchmarks or in small sample regimes due to their association with self-repair techniques OOlausoo al.) 2024 Chen
ct al. 2024 Shinn et al., 2023 Zhong et al. 2024).
In this paper, we systematically deconstruct the components of previous research on prompting techniques and propose unified framework for multi-tumn code generation. Our objective is to establish a comprehensive and strong baseline, and to explore behavior and limitations across various sample regimes. Our focus on competition-level coding benchmarks and sample budgets is motivated as follows: (1) Popular methods such as chain of thought (Wei et al. 2022 CoT) yield improvements on reasoning-hean tasks. However, they are designed to elicit reasoning traces for maximizing single-turn performance and are not inherently multi-turn. Competition-leve benchmarks require algorithmic reasoning and thus provide ideal testbed to evaluate whether CoT techniques can be extended beyond singlettrrn reasoning.
(2) Recent studies suggest that the performance gains from self-repair often modest when considering their generation cost (Olausson et al.) (024) and that repeated single-turn sampling serves as a strong baseline (Brown et al. 2024). As such, the trade-off'
"Equal contribution.

---

Published as a conference paper at ICLR 2025
Code
Problem
<Codex
<Code>
may fint, (ipast) (plis1)
voeds (Leput)) atteipp veeds sort thapelantdd xt [s-ond(a ardiet5311 wlse (ord
caspecaa)
if fails
Traceback
-odd(a1111) doe an sasgect, aea)3)
8
Problem Reason,
Instruct. Prompt
Inst. Response
<Code>
Reason. Response
+Coda>
custom__ortt
Caston socting kay ftr aac desc erding sodac. liple where firs alemant tiple of chammcteess odl indioss sscanding order and the secrnd alumenti tuple che weters moen indices desconding orcer.
Prompt
if falls
This funation
Feedback
Resson, Responsa
Problem Statement
Reason. Prompt
Inst. Response
and given assignment fiy up booksieef numbered from
book
Willi funcsions
def costen_aresa)::
iile
with
Inst. Prompt
the book. eulpographically axcindingg while Budi wants sod bestographielll
And
out a)ia
diocarding Setting their fight, Fayy decited to combine
their idea and sort ine-dssosnndnggs
Wilte comment of before each Fie
length of eecs sitte oel (ast, t) -atrttp (ppaz())
Execution Feedback
Exompte
titie and store oveings [iepett 13 atrepel for casgeie13
Insul
coce lowing testo: Input
Outbul
Trassbacki
"aac "asc-deec-endii -ed--dal--edddgly" critacis
Your wrong)
Figure 1: Our framework for evaluating LLM multi-turn code generation techniques. Top: In the default multi-turn setting, given programming problem, the model generates a code solution, interacts with the runtime environment to gather execution feedback and retries in case of failure. Bottom: On top of the default setting, we gather reasoning (Reason.) prompts, instruction (Inst.) prompts, and execution feedback prompts. The problem statement is augmented with a reasoning prompt. After generating an answer to the reasoning prompt, an instruction prompt determines how program code should be generated. The execution feeeuack prompts vary in granularity, ranging from binary pass fail indicator to detailed tracing information.
between single-turn and multi-turn approaches, and the optimal allocation of resources between them, remains under-explored.
Our framework (Figure I enables mix-and-match combinations of single- and multi-turn code generation and chain-of-thought (CoT) techniques]] prompts that induce reasoning, such predicting problem attributes or writing natural language solutions first, and instructions that prompt different programming styles such as including comments or helper functions. Finally, we integrate execution feedback from intermediate solutions to allow for code repair. We conduct a comprchensive experimental survey across different benchmarks, L.M families and sizes, as well as sample regimes. Our analysis yields several key insights:
1. In the single-turn setting, combining reasoning prompts and instruction prompts achieves the best performance, and is more beneficial on larger models harder problems. We also identify CoT's that degrade performance (Section 5.I).
2. The multi-turn setting alone brings modest gains and sometimes worse than its single-turn counterpart under equal sampling budgets. The combination with CoT provides a significant performance boost on all models we study. Interestingly, detailed execution feedback prompts do not always translate to improved performance (Section 5.2) We show that this can be attributed to reduced diversity of generated programs which results in performance drops for large sample budgets.
3. LLMs can be instilled with reasoning behavior by finetuning on multi-turn CoT data (Section 5.3 The resulting model surpasses our best prompting configurations even without explicitly asking for CoTs during inference.
We use the term "chain of thought" to refer to a broad family of prompting methods eliciting intermediate steps before during code generation.
2

---

Published as a conference paper at ICLR 2025
2 BACKGROUND
2.1 SINGLE-TURN VS. MULTI-TURN GENERATION: PROBLEM SETTING
We assume coding problem D {s, u, f), where is the problem statement natural language (e.g. see Figure D. is a set of public tests, and i is set of private tests. A given code sample is considered correct if it passes all tests, or incorrect otherwise. Let æ denote an LLM that is able to produce code sample for D from user prompt p which includes the problem statement In the single-turn setting we thus obtain a code sample c T(- p).
In multi-turn code generation, we can distinguish between a Catural-Language-tto Code (NL Code) task in the first turn and Code-to- Co-ee (Code + Code) generation in subsequent turns. For a given problem, we generate a sequence of intermediary code samples c1,..., cr rather than just one. After each turn i, the code sample c; is fed back into the model together with execution feedback prompt to obtain the next sample Ci+1. This process is repeated T times until we either pass all public tests or until a maximum number of turns N is reached. More formally, we can obtain every intermediary sample c;, including the final code solution cr, as follows:
G ~ π(. Pl. C1 , ...1P......PPP... G.1.
In this setting, the first prompt pi the initial user prompt including the problem statement, and each for i >> 1 is execution feedbuck prompt containing the runtime result with error information or traceback optionally attached.
In the remainder of this study, the sequence (P1, CI..... ) is denoted trajectory, and the final code sample er is called the submission. Only the code sample eT is tested against the private tests t for correctness (i.e. intermediary code samples will only be tested against public tests u). Note that we sample not just one but several trajectories in parallel, starting with the same initial prompt p1.
2.2 EVALUATION METRICS
Pass 010
We are interested finding correct solution to a given pro- gramming problem with a fixed budget, i.e., with fixed number of code samples. For estimating the success rate of generated code samples, pass @ k is a widely used metric (Chen et al. 202T). For problem P and given budget of k: samples, @ l is the expectation that at least one sample is correct, i.., that passes all tests.
50
48
48
42
40
38
36
Limitations of @ k Pass@k k ignores computational re- quirements and thus puts single-tun tuglee-turnaa evaluations at a disadvan- tage. In multi-turn settings, solutions generations (i.e., LLM calls) and hence at a higher cost, ren- dering these two setups not directly comparable (Kapoor et al. 2024).
34
51.5
Pass 10:5100
obtained via several
51
50.5
50
In this study, opt to measure performance via pass n@ k LLi cet al. (022) rather than pass@/ k for a fair comparison of tech- niques. Pass n@k estimates the success rate of model on problem P using i generations but at most n submissions; is the expectation that out of submissions one of them correct (Appendix A). Following [i et al. (2022), we select submissions based on public test performance. Note that for k. both metrics equivalent. For each benchmark, report the average pass n @ or pass@* & over all problems.
49
10
Max Turns
Figure 2: Scaling number of turns not compute optimal. Pass @ 10 (Top) and pass 10@ 100 (Bottom) on CodeContests test set when in- creasing the number of turns with Llama 3.1 70B.
Figure 2 compares pass@k & and pass n@k when measuring performance a multi-tumn setting. Pass@10 10 (Top) keeps increasing if we increase the maximum number of turns. How- ever, pass 10@ 100 (Bottom) shows that compute optimality is lost after turns. Given a budget of 100 samples with 10 programs selected as submissions, the optimal allocation of compute is obtained
3

---

Published as a conference paper at ICLR 2025
SINGLE-TURN
MULTI-TURN
Reasoning Prompts (NL - NL.
Prompts
Feedback Prompts
(NL - Code)
CoA [1], Self-discover [2]. Decomposed prompting [3)
Parsel [4], LLM assisted code cleaning [5]
Learn Debug (LDB) NeXT [7], Self-Debugging (9)
CodeChain [9), Self-refine [10), REFINER (11], CodeRepair [12),
General Specific Prompting [13]
Natural Language
Code
Execution Feedback
Explain inpul output pains
Generate naive solution.
Your code wrong.
Rate the probiem dificully
Proposs clever algoritim.
You led tests. Expected 5,
aclution language.
nacessary intemaaiste veriables
You failed the tests: for input expected: 5 got: You passed some seats: for inpul expected: and got:
Rellect on the protlem, describe
Weite the docatring and signatures helper functions
Here
analyzo fo your code.
Figure 3: Prompting space explored in our survey. We explore chain of thought prompts at three different levels: before the first code generation (reasoning prompts), with code generation (instruction prompts), and after the first code generation (execution feedback). The corresponding works from the single-ture and multi-turn reasoning and code generation literature are: [1] Gao et al.
(2024), [2] Zhou al. (2024), [ al. (2023), [4]Zelikman al. (2023), [Jii
(2024b),
Zhong et al. 2024), ), [7] et al. (2024), [8] Chen et al. (2024), [9]
Le et al. (2024). [10] Madaan
al.10 (2024).
Paul al. (2024). [12] Tang et (2024)
[13] Li (2023a).
by generating trajectories with at most. As such, throughout this paper, we favor pass @@k and report pass@ &t only when comparing single-turn results exclusively.
PROMPTING AND FEEDBACK SPACE
We map the space of prompting techniques studied in experimental survey in Figure As CoT can intervene at different times code generation, we categorize reasoning prompts (NL - NL) that elicit understanding of the problem before code generation, and instruction prompts (NL - Code) that guide the code output to enhance readability and modularity. These prompts can be applied in single-turn and multi-turn approaches.
In the multi-tumn setting, we also introduce execution feedback prompts directly harvested from the runtime environment, serving as additional information for the model to self-repair within turns. We aim to determine the type of feedback which most effective on competitive programming benchmarks the large sample regime. We thus evaluate several types of feedback, ranging in granularity:
Binary feedback: A simple pass/fail indicator.
Failed tests feedback: Provides expected and actual values for failed unit tests, along with tracebacks if any runtime errors are encountered.
Failed & passed tests feedback: Expands failed tests feedback by also including input/output information for passing tests.
LDB feedback (Zhong al. 2024): Offers debugger information, printing intermediate variable values and separating the code into blocks. The model must identify at which block the code failed and attempt fix
CoT and execution feedback are incorporated into the generation through specific prompts as illus- trated in Figure [ [ (Bottom). As we will show in Section different types execution feedback induce different multi-turn behavior that can be classified either exploratory exploitative.
4

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTAL SETTING
Models We perform experiments with the Llama Instruct series of LLMs, including Llama 3.0 and 3.1, 8B and 70B models (AI @ Meta 2024). We use Llama 3.1 405B and GPT-4o in small sampling regimes only due to compute constraints.
Single-turn Our grid search comprises 8 reasoning prompts and 6 instruction prompts, detailed in Appendix G The reasoning prompts elicit intermediate steps either in natural language or with partial code. The instruction prompts either increase code readability ("describe"), break down the solution into modular code ("modularity"), "). or bias the type of solution (" ("solution"). Although we perform one more step of LLM inference for the reasoning prompts, we do not consider it an additional turn as our study compares the number of code attempts per problem and the effect of adding different types of extra tokens. We argue that this is equivalent to single LLM call which groups all the reasoning prompts together, modulo the number of LLM forward passes. We generate with nucleus sampling (Holtmman et al., [020] top-p=0.95) and a temperature of 1.0 encourage output diversity.
Multi-turn When performing multiple consecutive attempts at solving a coding problem, we set the code attempt limit to 3; this motivated by the multi-turn results in Section 2.2 which reveal three turns as compute-optimal. We take the best reasoning prompts from the single-turn setting and combine them for up to reasoning steps before code generation. We also introduce the CoT-retry setup, which allows for adaptive inference budget based on problem difficulty. In the first turn, we omit CoT prompts. If the first solution fails on more challenging problems, we prompt the LLM with combination of execution feedback and reasoning prompt. We employ a different prompt for each turn (see Appendix G.3). We also ablate different granularities of execution feenttionfeedbbr feedback. We do not include CoT prompts in this feedback comparison to isolate the effect of different feedback types.
Rejection Sampling Finetuning With the Llama 3.1 70B model, we use the Co'I-retry strategy to generate 3-turn trajectories on the CodeContests training set. We filter out trajectories with incorrect final code and perform supervised finetuning on the resulting data (details in Appendix B.2.
Benchmarks We conduct our experiments on two competitive coding benchmarks in the zero-shot setting: (1) CodeContests (Li et al. 2022) contains 13k programming problems in the training set and 117/165 problems in the valid/test set. Each problem contains public tests, private tests, and generated tests. We use public tests to provide execution feedback in the multi-tun celli-turnn setting and use all available tests to evaluate the final submission. (2) TACO (Li et al. 2023b) is a collection of problems sourced from CodeContests, APPS (Hendrycks et al., 2021), and various programming contest platforms. The test set is split into distinct difficulty levels: easy, medium, medium-hard hard, hard, and very-hard. with each level comprising 200 problems. This stratification allows to examine the performance of different prompting strategies across difficulty levels. We use the first test case as the public test.
RESULTS
In this section, Table I and 2 first present maximum model performance for specific CoT variants. We then conduct a series of detailed experiments to better understand the performance impact of individual prompting methods. We structure our presentation by key findings outlined Introduction.
5.1 SINGLE-TURN SETTING: CoT WORKS BEST FOR HIARD PROBLEMS, LARGE MODELS, IIII SAMPLING
We first investigate the impact of various CoT prompting strategies on models in the single-turn setting. There will be no execution feedback prompts. Therefore, our grid search involves searching the space of reasoning prompts (NL + NL) and instruction prompts (NL - Code).
Reasoning and instruction prompts can work together. We first compare the effect of various reasoning prompts, instruction prompts as well as combinations of both. Synthesized results are presented in Table 3 and we refer to Appendix C. for the complete set of experiments that led to 3 An interesting observation is that even the best performing reasoning and instruction prompts for pass @ 100 can decrease model performance in small sampling regimes (pass @ 1). Although reasoning prompts provide larger gains than instruction prompts (with the exception of Llama 3.1 70B), combining both results in the best performance.
5

---

Published as a conference paper at ICLR 2025
Table 1: Up +10% pass @ with multi-turn CoT on CodeContests test with high lemperature
(1.0) and large sampling budget. In the multi-turn setting, we use a maximum of 3 code attempts (i.e., 3 turns) with the "failed tests" feedback. The pass @ is calculated from 200 trajectories for both single-tum and multi-turn settings, We also report the pass rates for Llama 3.1 70B after Rejection Sampling Fine-tuning (RFT) (Section 5.3) Prompts are the same across sample sizes per model.
CodeContests / Test
Model
Variants
@@3
10@:30
33@ 100
100@000
Llama 3,0 8B
2.9
8.0
12.6
CoT
3.4+0.5
11.7-3.7
17.3+4.7
Multi-tum
2.4-0.5
8.0+0.0
12.8+0.2
16.7
Multi-turn CoT
2.8-0.1
9.841.8
14.9+2.3
19.4
Llama 3.0 70B
9.6
18.9
23.1
CoT
10.440.8
26.0
83.0+9.9
Multi-turn
10.140.5
21.0-2.1
26.7+3.6
32.7
Multi-turn CoT
11.141.5
26.5-7.6
34.3
40.4
Llama 3.1 8B
7.7
18.2
23.8
CoT
8.040.3
19.5-1.3
26.1+2.3
Multi-tum
7.0-0.7
18.8-0.6
24.5+0.7
30.4
Multi-turn CoT
6.9-0.8
19.1-1.2
26.0+2.2
31.5
Llama 3.1 70B
24.1
42.3
49.8
26.442.3
47.8-5.5
54.8+5.0
+ Multi-turn
24.1
43.8-1.5
51.6+1.8
56.2
Multi-turn CoT
27.7
48.4.6.1
55.3
59.6
Llama 3.1 70B
26.2
45.1
50.9
Multi-tum
29.7
50.5
57. 57.2+6.3
61.1
Table Benchmarking of CoT across models: GPT-4o and Llama. Pass @ (%) and pass 1 1@3 (%) with low temperature (0.2). As models become more capable, repeated sampling surpasses straightforward extension to multi turn (c.g. GPT-4o) or single-ture gurle-turn&no CoT (c.g. Llama 3.1 405B). A tailored multi-turn CoT, however, improves pass 1 @@3 performance across all models.
GPT-4o
Llama 3.1 70B
Llama 3.1 405B
Variants
I@1
1@3
I@1
1@3
1@1
1@3
Single-turn
17.0
27.6
27.3
27.8
32.9
+ CoT
25.5+8.5
29.0+1.4
25.5+2.5
28.941.6
25.1-2.7
31.8-1.1
+ MMuuliiturn
23.1
29.542.2
35.442.6
M.Muliittra CoT
31.5
31.5.4.5
40.1
Table Combining reasoning and instruction works best as compared to each individually for single-turn CodeContests test set (chosen based on pass @ 100 performance per model). In the best categories, results worse than the baseline are underlined.
Llama 3.0 8B
Llama 3.0 70B
Llama 3.1 8B
Ilama 3.1 70B
pass 100
pass@
pass@ 100
6
pass@ 100
pass@1
pass@ 100
Baseline
12.3
23.8
3.8
22.8
16.7
48.9
reasoning
15.6
Worst instruction
1.4
11.3
3.4
25.1
3.7
20.9
14.9
48.4
Worst Combination
13.2
43.5
Best reasoning
1.8
15.7
7.0
30.4
4.1
25.7
15.7
52.2
instruction
Best Combination
1.5
17.3
5.3
33.1
4.0
26.1
16.1
54.1
CoT is most helpful for large models. With the smaller Llama 3.0 8B and Llama 3.1 8B, observe from Table 3 that the best combination of reasoning and instruction prompts provides relatively small gains of 5.0% and 3.3% pass @ 100 the CodeContests test compared to the

---

Published as a conference paper at ICLR 2025
EASY
MEDIUM
MEDIUM HARD
HARD
VERY HARD
20
15.
25
passze
20
N
15
Worst CoT
Figure 4: CoT helps most hard examples. From set of reasoning and 6 instruction prompts commonly used on competitive coding benchmarks, we extract the pass rate the best and worst prompts amongst all 63 (8 for Llama 3.0 8B. We compare on different difficulty split of the TACO dataset. The relative gain from tailored CoT increases with problem difficulty and sampling size.
(6 + 1) combinations (including no reasoning or instruction)
improvements of 9.3% and 5.2% from the corresponding 70B models. Interestingly, we found that not all sets of prompts beneficial. the worst combination degrades the pass 100 of Llama 3.1 70B by up to 5.4%. CoT makes performance worse if the model fails to follow the instructions or makes the LLM propose a sub-optimal plan. Sub-optimal plans are usually brute force approaches to solve the problem which do not fit the time limits constraint (see Appendix for an example).
CoT most helpful for harder problems. With the TACO dataset, which provides a difficulty split, we can observe that CoT does help smaller models on harder problems. that the relative gain from the best reasoning and instruction prompt combination, compared with the baseline performance (No CoT), increases with problem difficulty. For example, the pass @ 100 of Llama 3,0 8B nearly doubles with CoT on the very-hard test split (2. 1%.1% + 3.9%). We show in Appendix C that this observation generalizes Llama 3.1 8B and 70B model.
Prompt efficacy is model and sample size dependent. No singular reasoning and instruction combinations work best across sampling sizes and models (see Appendix C.2 for detailed analysis). Reasoning prompts that simplify the problem (e.g., self-reflection, explain input-ouiput pairs) benefit smaller models (8B models) whereas larger models (70B, 405B, GPT-4o) gain most from generating parts of the solution (e.g., write function docstrings). "Solution" iifioon"-orr bbased instruction prompts are the most efficient across models, specifically for the Llama 3.1 series, as shown in Figure
Pass@1
Pass@ 100
A from baseline
describe
2
modularity
solution
Figure 5: Solution- based instruction prompts work best across Llama 3.1 models. We separate instruction prompts into "describe" (e.g., add comments, imports), "modularity" (e.g., add helper functions) and "solution"(e.. "(e.g., write naive solution, propose a clever algorithm). The performance difference (A) is normalized with respect to the baseline and standard deviation per pass rate.
5.2 MULTI-TURN SETTING: SELF-REPAIR LIMITED WITHOUT CoT AND PROPER FEEDBACK
We summarize our multi-turn results in Table With a fixed number of samples, i.e., k in pass n multi-turn alone provides modest gains only (usually less than +2%) and sometimes even reduces pass 1 @ performance compared to drawing independent samples in single-turn mode. Notably, this is the case for smaller models (Llama 3.0 and 3.1 8B). In this section, we take a closer look at performance drops in the multi-turn setting and explore methods that can take ad advantage of accessing previous wrong solutions.
7

---

Published as a conference paper at ICLR 2025
Llama3.1-80
Llama3.1-708
2.5
2.0
2.0
1.5
Denaty
1.5
Density
1.0
1.0
0.5
0.2
0.4
0.6
1.0
0.0
0.2
0.6
0.8
1.0
Similarity Scores
Similarity
binary feedback
failed test
passed feedback
feedback
Figure 6: Fine-grained feedback induces exploitative behavior: Distribution of consecutive code similarity scores within dialog for different types of feedback, obtained from Llama 3.1 8B and 70B samples (temperature 1.0). The higher the similarity scores between consecutive codes in the same dialog, the more the model exhibits exploitative behavior.
Reasoning prompts are not additive. It is tempting to consider that stacking more reasoning prompts before code generation will further guide the model towards cor- rect solutions. For example, prompts might increase the granularity of reasoning: self-reflect on the problem, ex- plain the input/output pairs, write helper functions, and fi- nally output a full code solution. However, we empirically find that across models, one step of reasoning provides the most sigmificant boost. The performance plateaus or even decreases with two or three steps. Increasing the number of reasoning steps hurts both Llama 3.0 and 3.1 models (see Table 7 in Appendix [D.T). For the best models, a single step with reasoning prompt is most beneficial.
Pass10g1.00
50
base model
base model+CoT-retry
RFT model
Max Turns
Figure Reasoning and execution feed- back prompts, and RFT, enhance both single- and multi-turn performance for Llama 3.1 70B.
CoT-retry works best. For Llama 3.0 models, simply tending the single turn reasoning and instruction prompts to the multi-turn setting yields superior performance (reported "Multi-tura CoT" in Table [ However, as models become more capable, an increasing number of problems in CodeContests are solved the first attempt without specific prompts. CoT-retry only reasons when the first attempt fails and therefore works best across Llama 3.1 models for all sampling sizes and benchmarks ("Multi-turn CoT" in Table D). Figure 7 decomposes its per-turn performance. When extending the number of turns from 2 to 3, Llama 3.1 70B alone shows diminishing gain while combination with CoT-retry still increases the performance by a large margin.
Execution feedback granularity determines exploration-exxloitatioo behavior. Given previous incorrect code and execution feedback, subsequent attempts consist of a fresh attempt (exploration) or of updates to prior solutions based on feedback (exploitation). We quantify this behavior by computing similarity scores between two consecutive solutions (details in Appendix(B.. [B. Figure 6 shows that with more fine-grained information provided via execution feedback, models exhibit exploitative behavior (high similarity scores). Exploitation be desired property on relatively easy problems where errors are due to simple bugs. However, we posit that diversity is key to improving performance on difficult problems, i.c., exploratory behavior within trajectory based on the execution feedback prompts. This matches our experimental results: simple execution feedback (e.g., binary, failed tests) provides optimal performance for most models (Appendix D.2p.
5.3 COT REJECTION SAMPLING FINE-TUNING: MODELS CAN INTERNALIZE REASONING
We investigate whether LLMs can benefit from finetuning on reasoning traces obtained via CoT prompting. We thus perform Rejection Sampling Finetuning (RFT) on Llama 3.1 70B, where the reasoning, instruction and execution feedback prompting strategies we consider act as policy improve- ment operators: they elicit the model's 's reasoning ability and produce a higher number of trajectories
feedbbaa
8

---

Published as a conference paper at ICLR 2025
Table 4: Multi-turn CoT and RFT generalize TACO test set. Pass @ (%) of Llama 3.1 70B on multi-turn TACO test set with temperature 1.0. We use the best multi-tur CoT found on CodeContests. We use the model RFTed on CodeContests training set (after decontamination, details in Appendix I1 and report its performance directly on TACO without COT.
Model
casy
medium
medium hard
very_bard
1@3
100@:00
1633
100@ 300
1 @63
100@ 300
1@3
1000 300
1@3
100 300
14.2
44.6
4.4
Multi-turn CoT
32.3
59.8
15.0
46.2
10.8
38.5
5.8
22.8
11.8
Llama 3.1 70B
34,1
58.9
18.0
45.3
13.0
39.4
8.1
23,3
3,5
12.0
Within Correct Trajectories
Within Incorrect Trajectories
250
200
Count
Count
1550
150-
10000
100
50-
500
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Similarity Scores
Similarity Scores
before multi-tum CoT RFT
after multi-turn CoT RFT
Figure 8: RFT makes the model produce more diverse code within trajectories as shown by the consecutive codes' similarity scores before/after RFT on CodeContests test set evaluated with multi-turn no COT. This shift towards more exploratory behavior contributes majorly to the gain of correct trajectories.
with correct submissions. Given the low variance across dif- ferent feedback types (Table in Appendix [D.2), we opt for simplicity and use the "failed tests" execution feedback com- bined with CoT-relry for data generation.
CodeContests/test
60.
pass k@3k
More specifically, improve model by collecting dataset of correct trajectories sampled from æ with CoT enabled inference time, 2) removing the CoT prompt in the collected trajectories, and 3) finetuning т with the standard next-token prediction objective. With this strategy, we can now obtain CoT- level trajectories without adding specific prompts at inference time.
40
30
Base Model. No CoT Base Model,
100
10' k
10a
Figure 9 Table and Table 4 show that the RFT model pro- vides additional gains over inference methods across sampling sizes and datasets. Beyond performance, RFT multi-turn CoT improves sampling diversity (Figure 8) and self-repair capacities, especially for long trajectories (Figure 77. Behavior- wise, we show in Table IT (Appendix F.T) that RFT results in model responses with increased textual content.
Figure 9: Llama 3.1 70B's pass @33 on CodeContests. CoT-retry increases the performance in large sampling regimes. RFT transfers this reasoning ability to no CoT set- ting and lifts the pass rate curve across sampling budgets.
RELATED WORK
Chain of Thought with Code Chain of Thought (CoT) enables step-by-step thinking for LLMs to solve mathematical word problems in either few-shot (Wei et al., 2022) or zero-shot (Kojima) ct al., 2022) settings. Many variants, e.g., Tree of Thought (Yao et al.) 2023a). a) have emerged in code generation since. Chen et al. (2023b) and Gao et al. (2023) translate natural language mathematical problems in executable code for the model to separate reasoning and computation. These methods rely on the LLM outputting correct code represent problem. We this work as tangential to ours as boosting LLM coding performance will also help on overall reasoning tasks. Higher levels of
abstractions (Khot et al. 2023 Zhoou et al. 2024
2023
Zelikman et al. 2023 Jain al. 2024h) and

---

Published as a conference paper at ICLR 2025
self-repair techniques (Paul et al. 2024 Li et al. 2023a: Ridnik et al. 2024) have been proposed. Beyond inference methods Wadhwa et al. (2024); ;; Yu et al. (2024): : Zelikman et al. (2022) Hosseini
et al. (2024); Pang et al. (024) explore
training algorithms and loss functions to learn from CoT.
In comparison, we bring novelty to the type of CoT used in training (multi-turn) and rely on simple
Rejection Sampling Fine-tuning (RFT) (Touvron et al. (0223 Yuan et al., [023 AT @ Meta 2024). It
has been shown to achieve good performance, with less data compared to SFT (Setlur et al. 2024).
Execution feedback Currently LLMs struggle to understand code execution feedback (Gu et al.
2024) as this type of data is rarely present in their training set. [Zhong et al. (2024) and Ni et al. (2024)
try to mimic "print debugging" convey intermediate code steps to the LLM. Olausson
(2024)
found that the effect of self-repair largely depends on the text quality of the subsequent reasoning and therefore use only textual feedback. our setting, we are interested in the feedback which could be
directly harvested from the execution environment. [Shi et al.] (2022); [i et al. ) (Chen et al.
(2023a) likewise proposed unit test generation as way to increase coverage with execution feedback. Adding test generation to our pipeline would be an interesting avenue for further work.
With the rise of LLM agents (Kapoor et al. 2024) and the scaling effect of test time techniques (Li et al. 2022. (Snell et al. 2024: Brown et al. 2024), inference optimization against compute resources becomes increasingly relevant. Similar to our pass n@k argument in Section 2.2 Kapoor et al. (2024) discuss the importance of controlling for generation cost in Al agent evaluations.
Inference Optimization
7 LIMITATIONS
In our multi-turn setting, we do not explore further branching at the second or third turn, i.e., more complex tree structures (Tang al.] 2024) or in general inference- frasee search approaches (Snell cet al. 2024). e.g., with look-ahead or backtracking, as we focus on the effect of additional CoT tokens generation, Although a maximally fair comparison (at the cost of complexity) should account for total input and output tokens (Olausson et al. 2024) as well as model size (Hassid et al.) (024). we believe pass n@k, k. which stresses the number of code attempts, constitutes a simple yet superior alternative to pass @assskk k. Our RFT is similar to Expert Iteration (Anthony et al.] 2017) and ReST (Gulcehre et al.) 2023) when considering single iteration only. We also assume trajectories with correct final code contain correct reasoning. Adding a Process-Reward Model (PRM) or "critic" LLM (Zheng et al., 2024) to rate and filter the correctness of the reasoning tokens could enhance training data quality and diversity. Future work could benefit from exploring more advanced inference techniques such as prompt tuning (Lester et al.] 2021) or training strategies such as including "near-correct" trajectories (Pang al.] [024: Setlur et al. 2024) with multi-turn COT. Finally, we speculate that the effectiveness of different prompts for different LLM families (particularly the Llama 3.0 vs. 3.1 series vs. GPT-4o) could be attributed to the mixture of finetuning data (Chung et al. 2022). Exploration of this topic is beyond the scope of this paper.
8 CONCLUSION
In this work, we present a comprehensive experimental survey on various reasoning, instruction and execution feedback prompts in the single-turn and multi-turn code generation task at scale. Our results on two competitive programming benchmarks, CodeContests and TACO, suggest that incorporating CoT techniques, originally designed for single turns, and execution feedback prompts into the multi-turn setting is non-trivial. Due to the difficulty of the benchmarks. major contributor to performance is problem understanding rather than the ability to perform code repair with detailed feedback. With set compute budget, using multiple turns alone can hamper performance compared to repeated sampling with high temperatures. Biasing the model with adapted CoT based on problem difficulty at each turn boosts its self-repair abilities and leads to consistent gains across all model series and sizes. Beyond inference methods, our RFT experiment shows that multi-tumt reasoning traces triggered by prompts can be internalized, which leads to advanced reasoning abilities. We hope that our findings motivate further research in more advanced multi-turn settings. One example is repository-Ievel code agents, where models interact with complex environments to gather feedback and extensive planning and reasoning capabilities are demanded.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
As our paper focuses on inference methods with existing models, the key components for reproducibil- ity are access to models, datasets, and prompt descriptions. All the models (except our tine-tuned RFT model) used in this paper are publicly available at the time of writing: Meta Llama 3.0 and 3.1 series are open-weight, and gpt-4o-2024-05-13 (GPT-4o in the paper) are available through OpenAl API. The two benchmarks we use: CodeContests phttpss///il thub. wwt.cooo//g/oe htmm//ooggeeeeeemidd//oooooooo/e deepmind/code contests) and TACO a complete list of all our prompts in Appendix G to reproduce single-turn and multi-turn experi- ments. We present the details of computing similarity score with normalization in Appendix B. Regarding finetuning, our main contribution relies on the data augmentation technique on Code- Contests for which
www.c/mm//aaa.ecc (amm///l0g0pe/TACO) are publicly available. We provide
present the details in data collection, deduplication, and decontamination approach, as well as statistics such as the number of trajectories and the number of total tokens in Appendix B.2 B.3 and [
We detail finetuning hyperparameters in Appendix [B.2 to reproduce our RFT model training. We will release the code for our multi-turn and CoT methods to facilitate reproduction.
[.2.B.3
ACKNOWLEDGEMENT
We thank Quentin Carbonneaux, Baptiste Rozière, Jade Copet, Olivier Duchenne, Fabian Glöeckle, Badr Youbi Idrissi, Nicolas Usunier, Sten Sootla, Chris Cummins, Sida Wang. Pierre Chambon, Matthieu Dinot, Ori Yoran, Kush Jain, Naman Jain and all the members in FAIR CodeGen team for helpful technical contributions, suggestions, and insightful discussions. We thank the Infra team for the support for enabling a seamless compute cluster experience.
REFERENCES
Llama Team Al @ Meta. The Llama Herd of Models, 2024.
Google DeepMind AlphaCode Team. AlphaCode 2 Technical Report. Technical report, 2023,
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. N. Vishwanathan, and Roman Garnett (eds.), Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5360 5370, 2017. URL https: http:htppps/prrrceedieeee newrins.ccppccc 7t5777/5565111442220066666776660
html
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint N.Ki::208.0073:: 2021.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, 2024. httpsssss////axii .rg www.:o////bb/////1// 217.2
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali. Rwanda, May 1-5, 2023. OpenReview.net, 2023a.
URL httpp:://pperrevie.. wwwp:/eeeeeoooe
Mark Chen. Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377 2021.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Trans. Mach. Learn. Res.. 2023, 2023b. URL htttps:/ www.:eewwnwtfooummmeeeemmmm htt.://oooruumm/ie///4tataaa
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In The Twelfih International Conference Learning Representations, ICLR 2024,
11

---

Published as a conference paper at ICLR 2025
Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.nett forum?id-KuPixlqpig
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai. Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction finetuneed language models. J. Mach. Learn. Res., 25:70:1-70:53: 2022. URI j/jmr or wwwggg/pap/p//////22 28700 html
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. Pal: Program-aided language models. International Conference on Machine Learning, pp. 10764-10799, PMLR, 2023.
Silin Gao, Jane Dwivedi- Yu, Ping Yu, Xiaoqing Ellen Tan, Ramakanth Pasumuru, Olga Golovneva, Koustuy Sinha, Asli Celikyilmaz, Antoine Bosselut, and Tianlu Wang. Efficient tool use with chain-of-abstraction reasoning. arXiv preprint arXiv:2401. 1. 17464, 2024.
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL URU/htpp://ppprreeieew.er
Alex Gu, Wen-Ding Li, Naman Jain, Theo Olausson, Celine Lee, Koushik Sen, and Armando Solar- The counterfeit conundrum: Can code language models grasp the nuances of their incorrect generations? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL. 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 74-117. Association for Computational Linguisties, 2024. doi: 10.18653/ V1/2024.FINDINGS ACC.7. URL hhttth:t//dooii org. 111..0.11163///22222222 findings-acl.7
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Amaud Doucet, Orhan Firat, and Nando de Freitas. Reinforced self-training (rest) for language modeling, 2023. URL http://aaxvvvorg//aee/
08998
Michael Hassid, Tal Remez, Jonas Gehring. Roy Schwartz, and Yossi Adi. The Larger the Better? Improved LLM
via Budget Reallocation. arXiv:2404.00725 [cs]. Mar 2024.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measur- ing coding challenge competence with APPS. In Joaquin Vanschoren and Sai-Kit Ye- ung (eds.), Proceedings of the Neural Information Processing Systems Truck on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021. virtual, http:///tttttteteeeeeeemersrreee neeeeddininnnnnnnnnneeee ips cc/ http:/apee///// 2222/h//221/hhsh/ 2021. htt.eeeeeeeeeeeeeeeeaeee2e222ee0 hbtp::/o2aao000bbbbbaartttrrrrr html
c24cd76e1
David Herel and Tomas Mikolov. Thinking tokens for language modeling, 2024. URL https //arxiv. wtt.:/r//abb//25..066666
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. ICLR. OpenReview.net, 2020.
Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprint 2xXiv:2402.06457, 2024.
Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. MapCoder: Multi-Agent Code Generation for Competitive Problem Solving. arXiv:2405.11400 [cs], May 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint 20070:2403.07974, 2024a.
12

---

Published as a conference paper at ICLR 2025
Naman Jain, Tianjun Zhang. Wei-Lin Chiang, Joseph E. Gonzalez, Koushik Sen, and Ion Stoica. Lim-assisted code cleaning for training accurate code generators. The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net, 2024b.
Sayash Kapoor, Benedikt Stroebl, Zachary S Siegel, Nita Nadgir, and Arvind Narayanan. Ai agents that matter. arXiv preprint arXiv2240.001502 2024.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: modular approach for solving complex tasks. In The Eleventh International Conference on Learning Reprexentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, OpenReview.net, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Sanmi Koyejo, Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL /ttptersppppppp...t.. nips wtwpppppppppppppppeeeetpeepppppe cc/paper_pppppeceeecce files/ paper/20222 2022/hash2hhhaaee hbbbh/9bbbbbb99
Hung Le, Hailin Chen, Amrita Saha, Akash Gokul, Doyen Sahoo, and Shafig Joty. Codechain: To- wards modular code generation through chain of self-revisions with representative sub-modules. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Aus- tria, May 7-11, 2024. OpenReview.net, 2024. URL https:://penrevview. net/formm?d= wYhglxSj8j
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-300100 3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.166553w122222.emmpmaaaaa //acllantollogyoor//220 http::/oogg..rgg2222..memmmmmmam wwn.p-main........mm.m...mmm.amm 243
URLhhttss
Jierui Li, Szymon Tworkowski, Yingying Wu, and Raymond Mooney. Explaining competitive-level programming solutions using llms. arXiv preprint arXiv:2307.05337. 2023a.
Rongao Li, Jie Fu, No-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset, 2023b. URL htthh::t///aarr org/
abs/2312. 14852
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittieeser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition- code generation with alphacode. Science, 378(6624): 1092-1097, 17966624):0022-109, 2022.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, and Pengcheng Yin. Next: Teaching large language models to reason about code execution. In Forty- first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
httt::://pperreviinee U
htlphhtps:///
Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. Is self-repair silver bullet for code generation? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL http://rwwwweeeeeeeoo/oomm////nd
OpenAl. Gpt-4 technical report. arXiv:abb/22033.877. 2023.
13

---

Published as a conference paper at ICLR 2025
Richard Yuanzhe Pang. Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason poeferenceooptmiiiiii optimization. arXiv preprint 2XXi::2404117333 2024.
Weston. Iterative reasoning preference
Debjit Paul, Mete Ismayilzada. Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. REFINER: reasoning feedback on intermediate representations. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024 Volume I: Long Papers, St. Julian't s, Malta, March 17-22, 2024, pp. 1 1100-1122 Association for Computational Linguistics, 2024. URL https://aclantholoog wrg.2/2......2... eacl-long. 67
Tal Ridnik, Dedy Kredo, and Itamar Friedman. Code generation with alphacodium: From prompt engineering to flow engineering. arXiv preprint arXiv:2401.08550 2024.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2024. URL https///arriv.org//bb22228 12950
Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. RI on incorrect synthetic data scales the efficiency of Ilm math reasoning by eight-fold, 2024. URL
14532 14532
Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I. Wang. Natural language to code translation with execution. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. EMNLP 2022. Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 3533- 3546. Association for Computational Linguistics, 2022. doi: 231
231. |Rt htttps:///ol: http:///ooooorg///// 10. 18653/v1/2022.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurlPS 2023. New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: //pp:e://ppppse/e nips cc.ppprrrcppprpp.ppppppppppp httpsc/apeeeiills/ppprrrrrr filess/aperr/2233/ 00444b878b0788669544488833 hatpaa/eeeeeeeettaaaaarr--eerree
html
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024. URL URL/httss://arxiw.rrr 03314
Hao Tang, Keya Hu, Jin Peng Zhou, Sicheng Zhong, Wei-Long Zheng, Xujie Si, and Kevin Ellis. Code repair with llss gives an ppploratiio-exploitaaaee tradeoff, 2024. URL httpps///axxv org/ hrggabb//405.. 17503
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami. Naman Goyal, Anthony Hartshorn. Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and tnnettned chat models, 2023. URL URI //arxiv...vv wwi.iivvoogg/// org/ httpg/bb//300... 09288
14

---

Published as a conference paper at ICLR 2025
Somin Wadhwa, Silvio Amir, and Byron C Wallace. Investigating mysteries of not-augmented distillation. arXiv preprint arXiv:2406. 14511, 2024.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models, Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neu- ral Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA. USA. Decem- ber 10 16. 2023, 2023a. URL /ttt::://ppprr..iiiiiii httpiipps.cppppprrraaee pcrpppperfllllspppperreee hatpp//r/2////hhhh 271 17992222b8 F
html
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023b. URL http:tttt////peeeee///e//
net/ www..eee/ooommiddWe.lllllll
Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system 2 into system 1, 2024. URL
https://arxiv.org/abbb// 06023
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models, 2023. URL /ttpss::s///axxi org/ abs/ 2308.. 2tt.///3....22222../. 01825
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances Neural Information Processing Systems, 35:15476-1546
2022.
Eric Zelikman, Qian Huang, Gabriel Poesia, Noah Goodman, and Nick Haber. Parsel: Algorithmic reasoning with language models by composing decompositions. Advances in Neural Information Processing Systems, 36:3 1 -3523,,,,, 2023.
Xin Zheng, Jie Lou, Boxi Cao, Xueru Wen, Yuqiu Ji, Hongyu Lin, Yaojie Lu, Xianpei Han, Debing Zhang, and Le Sun. Critic-cot: Boosting the reasoning abilities of large language model via thain-of-thought, critic. arXiv preprint arXiv:2408. 16326, 2024.
Li Zhong, Zilong Wang, and Jingho Shang. Ldb: A large language model debugger via verifying runtime execution step-by-step. arXiv preprint arXiv:2402. 16906, 2024.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe V. Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
URL htttttsss//pppereviiew wwt.t//oroumiiddi///00055.
Pei Zhou, Jay Pujara, Xiang Ren, Xinyun Chen, Heng- Tze Cheng, Quoe V. Le, Ed H. Chi, Denny Zhou, Swaroop Mishra, and Huaixiu Steven Zheng. Self-discover: Large language models self-compose reasoning structures, 2024. URL https: //aaxss arg//bb/2402...o.mm 03620
15

---

Published as a conference paper at ICLR 2025
A FORMULA AND ALGORITHM FOR PASS @ K METRICS
Formally, let N be the total number of code samples. Let F' be the number of codes filtered by public tests, among which there could be false positives. Let C' be the number correct codes that pass all the unit tests. The pass n @k for benchmark of problems is defined as follows:
pass n@k Problemss
1
Σ ==
(I)
where np = min(i, n).
Explanation The first term (ME))) is the probability of having i filtered solutions among solutions, which obeys a hyper-geometric distribution, HYPERGEOMETRICCF, N W-F, F- k). Given the number of submissions np min(i, n). the second of the correct solutions.
the probability of having none
In evaluation, instead of computing the combinatorial number, Monte Carlo estimation by re-sampling k solutions Thoor times for bootstrapping (in our case, we use 10000). The algorithm for such is described detail in Appendix A.3 of the Alphacode paper (Li al. 2022).
B REJECTION FINE-TUNING EXPERIMENT DETAILS
COMPUTING SIMILARITY SCORE
We compute the similarity score of two Python code snippets follows.
First, we pre-process the code snippet to remove formatting and variable naming effects. We normalize variable names by rumning in-order indexing scheme the Abstract-Syntax Tree (AST), shown in Figure [0] followed by simple formatting by lambda x: unparse (ast. px)se().) We note that there are 1 1%-2% of codes failing the parsing because of syntax error, in which case we skip this normalization step.
total_n = int ()nput()) for zange (total_co)
= int
v_1 range (v_0):
5_ nun, P. num list (int, spplitt)))))
(v_2, v_3) 1ist (map (int, ()puu()..plittt ())1
Figure 10: Example of variable renaming AST pass.
Second, we use difflib. SequenceMatcher to compute the similarity score for the normalized snippets.
B.2 RFT DATA COLLECTION
Our data collection pipeline consists of major steps: generation, filtering and post-processing, deduplication and decontamination. We present the details of each step, including the parameters we use and the dataset statistics.
B.2.1 GENERATION
Using CoT-retry. we generate 200 multi-turn trajectories with maximum of 3 code attempts using Llama 3.1 70B for each problem instance in CodeContests training set. The generation is in the standard chat format for Llama 3.1 series" We do not include the system prompt in the dialog. We use nucleus sampling (Holtzman et al. 2020 with top-P=0.95 and temeprature 1.0.
hhtpps://wee www.aa.w....c..........e.. htwm//doc/modeee-ooom
and- anwpddd--dd--rprpppptt -at--aadtpppptt-oa-aaaaaa-a---ee wow.tttfooattttaaaaamaa
16

---

Published as a conference paper at ICLR 2025
We follow the same format as the evaluation: Final code solutions are tested against all the tests, and the code solutions the middle of the dialogs are tested against public tests. the model solves the problem in the first turn, the trajectory will still be collected while there will not be execution feedback.
B.2.2 FILTERING AND POST-PROCESSIAN
After filtering the incorrect trajectories, we keep only 60% of all the generated trajectories where the code in the last turn passes all the tests. We assume that correct final code correlates with correct reasoning in the CoT and self-repair techniques. The of successful trajectories contains solutions to 7238 problems in the CodeContests training set (in total 13213 problems), among which 1105 problems are only solved under the multi-turn setting. Interestingly, we found 485 problems which could be solely solved under the single-turn setting of all the generated 200 code trajectories.
We apply additional post- post-processing to the trajectories by removing the CoT prompt introduced but keep the model response untouched. This enables the model to develop inherent CoT--like reasoning capabilities through fine-tuning.
We separate the successful trajectories into 2 sets: single-turn trajectories and multi-turn trajectories. The single-turn trajectories contain 426952 trajectories, solutions 6133 problems. The multi-turn trajectories contain 226382 trajectories, solutions to 6753 problems.
B.2.3 DEDUPLICATION AND DECONTAMINATION
We conduct L.SH-based deduplication on each set to the code solutions per problem instance to maximum of 50 solutions, by following the practice of Jain et al. (2024b). We use hash size 64, jaccard threshold 0.5, number of bands 60 and band size 5 for the LSH-based deduplication configuration.
We further conduct decontamination between the collected solutions and TACO test set (details in Appendix (1). This enables direct evaluation of the finetuned model on TACO test set to measure the generalization to TACO.
After deduplication and decontamination, harvest 177475 single-turn trajectories (in total 143M tokens) and I 60600 multi-turn trajectories (in total 285M tokens).
B.3 FINETUNING SETTING
We perform self-supervised fine-tuning the above-mentioned multi-turn trajectories using Llama 3.1 70B. We use standard cross-entropy loss on the last full body the model response in the last turn and treat all the previous user and model messages the prompt part.
The finetuning uses learning rate 2c 545 steps of gradient updates, sequence length 8192, global batch size 524288 tokens. We use AdamW the optimizer with weight decay 0.1, 31 0.9 and 32 = 0.95. The learning rate schedule cosine scheduling with 10 warmup steps annealing to 10% of peak learning rate the end of the training. We do not do early stopping evaluate the model checkpoint in the middle of the finetuning. Instead, our evaluation always compares model checkpoints under different configurations at the end of the funetuning.
The end-to-end finetuning takes 170 H100 hours with Tensor Parallelism of size 8 and Fully Sharded Data Parallelism (FSDP).
B.4 GENERALIZATION OF RFT MODEL
Beyond competitive programming tasks such as CodeContests and TACO, we studied whether our RFT model, Llama 3.1 RFT fine-tuned on CoT and multi-turn data generalizes to other code generation tasks. Table 5 and Table show respectively. For single turn, we report performance on the following code generation benchmarks: HumanEval+ (Chen et al. 2021 Liu et al.) 2024), MBPP+ (Austin et al. 2021 Liu et al. (024) and LiveCodeBemch-vá Jain et al. 2024a). We also report multi-turn performance on LiveCodeBench- Our RFT model performs similarly, sometimes with slight performance degradation, and often better
show results for the single-turn and multi-turn experiments,
17

---

Published as a conference paper at ICLR 2025
than Llama 3.1 70B, which shows that the model does not overfit to CodeContests and generalizes to unseen code generation benchmarks.
Table RFT model fine-tuned on CodeContests generalizes to other code generation datasets.
Each line corresponds single-turn performance evaluated without CoT prompts for both models.
Results are reported under the format pass@1 pass@ 10. We use temperature 0.2 for sampling.
LiveCodeBench v4
Model
HumanEval+
MBPP+
Easy
Medium
Hard
All
Llama 3.1 70B
71.8/ 77.0
65.2 70.9
73.8 / 85.0
22.0/ 37.4
3.3/ 7.2
34.2 45.3
Llama 3.1 70B
72.1 76.9
63.5 69.2
76.2 85.7
22.0/ 37.0
3.5/ 8.0
35.1 45.3
Table Better low sampling multi-turn performance with the RFT model. We prompt models without CoT and perform multi-turns with maximum of 3 turns. Results reported under the format pass 1 @@3 pass 10@30. We use temperature 0.2 for sampling.
LiveCodeBench v4
Model
Easy
Medium
Hard
Llama 3.1 70B
82.8 94.3
30.8 / 49.2
4.77 / 99.55
Llama 3.1 RFB
86.0 94.4
31,5 / 50.1
4.74/9.19
C ADDITIONAL SINGLE-TURE RESULTS
C.I GRID SEARCH RESULTS
We provide the complete grid search results for all our reasoning and instruction prompts across all models and pass rates for the single turn setting. This demonstrates the variability in effectiveness per sampling size and LLM series. The "weak solution" instruction prompt is clear winner for larger sampling sizes k 2 10. We show in Figure 11
12 13 and I4 the grid search of all reasoning and instruction prompts for the Llama 3.0 and 3. series. As we increase the sampling budget, we increase the sample diversity and the recall across all COT. For low sampling budget, most prompts hurt performance. CoT is the most effective with Llama 3.0 70B.
CodeContest Test Set LLaMA 3.1 8B
Pass@1
Pass@10
Pass@100
clever alporithm
weak
func. docstring
comment for Ince
check constraints
-2
no Instruction
amii seifreffection exclain pairs probium tog
No CaT
probiem aificuity
multiple solutions witte help. decaring
solution
NO Car delf seffruffaction
probism tog probiem dificuity
pairs
colution mutiiple solutions
writee heip. decaring
No Cat detf reffectiin explain pairs probiam tog
probfen dificuity
mutippe solution write help decarrng
calition
Figure 11: Grid search of all reasoning and instruction prompts for Llama 3.1 8B.
18

---

Published as a conference paper at ICLR 2025
CodeContest Test LLaMA
Pass@1
Pass@10
Pass@100
algorithm
weak solution
func. docstring
comment for Inc
he functions
no Instruction
-
No cot seif-eftec eef-refleciann
explai to Pairs prodiem top
probhem aiffcclty Hi. splutton
muttiple solutions write help decating
Cat self enffreffeccion explain tD Dairs prodiem top prothem diffcclty Hil sstition muttiple solutions wine help decatring
CaT sall retrreffcctinn explain tD Daira grobeem tag
prottem dificulty AiL natusion mutiipe solutions wrme help decatring
Figure 12: Grid search of all reasoning and instruction prompts for Llama 3.0 8B.
CodeContest Test Set LLaMA 3.1 70B
Pass@1
Pass@10
Pass@100
clever algor
weak solution
func, docstring
check constraints
use he per functions
NO Belkreflcctoon reteecttion
explinn 10 Part
probism prosten dificutt BL patution
multlee calutone
help, docaring
NO CoT bolk aektnflaction expeinn to DArs probiam proteem dificulty BRt palution
mullipe colutions
help, docatring
NO CoT mellection
expeen parr probiam protiem diffcuts Bit salution mulliple colutions
write help, doccsring
Figure 13: Grid search of all reasoning and instruction prompts for Llama 3.1 70B.
CodeContest Test Set LLaMA 3.0 70B
Pass@1
Pass@10
Pass@100
clever algor thm
weak
func. docstring
comment for line
check constraints
functions
no Instruction
anti infretcctiion tion
No GoT exclain pairs
probsen tog
problem dificuity
multipe solutione
wrsee help decating
solution
to CoT antt rerrrtccciion exclain cairs protsem tog problen dificitty
mutiipl solutions write help docatring
colution
No CoT dett rerrraacciion explain pairs protiem teg bretten dfficcityy
mutiple solutions
write heep decatrigg
colution
Figure 14; Grid search of all reasoning and instruction prompts for Llama 3.0 70B.
19

---

Published as a conference paper at ICLR 2025
CodeContest Test
Pass 1@3 Single Turn
gpt40
LLaMA 3.1 8B
LLaMA 3.1 70B
LLaMA 3.1 405B
algorithm
func. docstring
comment line
check constraints
use he per functions
-5
na instruction
celforeletten feetonn probiam
teplaan 50 pain
problem aircutty musiple sonttons help decuming
sulf-refaction bxpiain tO paice groblee problim clmsuit Ht esuntonn mulfile help ducctring
selfogellctee explain patro probiem problem dlfficuty
colution mutiipl satutions help decsoring
seff-reftecton explain frabien probiam simicoyy cabution mulippe nptutione dicctren
poirs
Figure 15: No gold CoT across models. Based on our grid search of instruction and reasoning prompts, compare all 63 single-turn results across three different models. With low sampling budget, most prompts perform similarly, if not worse than the baseline performance (without CoT). The best prompt (in green) differs for each model, but see similar patterns in the Llama models.
C.2 DETAILED ANALYSIS SINGLE-TURE PROMPTS
When comparing reasoning and instruction prompts, the values normalized with respect to the baseline in each respective pass rate specifically: x +- x- baseliiee
std(x) The value at O, therefore,
corresponds to no reasoning and no instruction prompts. We provide further results aggregated across models and types of prompts.
As demonstrated by Figure and Figure 17 we have large variations across models and prompt types and observe that no reasoning and instruction prompt always performs above the baseline. As shown in Figure 18] the best combinations often rely on "weak solution" instruction but vary across sample sizes for reasoning with "self--efleccton" for lower sampling budget and "helper functions" for higher sampling budget. We observed writing intermediate variables before code often made performance worse and could be qualified as the "worst" reasoning prompt for all models.
Instruction Prompts
Pass@1
Pass@:100
clever algorithm comment for line func. decstring helper weak solution
A from basalino
14
Figure 16: Group by instruction prompts averaged across all reasoning prompts for the Llama 3.0 and 3.1 models. We observe that "check constraints" is winner for pass@1 1 and "weak solution" for pass 100. Overall, "add comment before each line" seems the least efficient across models.
C.3 GENERALIZATION OF SINGLE-TURN BEST COT TO LLAMA3. MODELS
We show in Figure that the best CoT (i.e., reasoning and instruction prompt and their combination) found with Llama 3.0 8B TACO could be directly ported to Llama 3.1 8B and 70B models. We also observe that CoT brings more boost on harder problems by comparing the relative gain of pass rate the easy and very-hard split.
20

---

Published as a conference paper at ICLR 2025
Roasening Prompts Across LLaMA se Models
Pasag1
Passgg100
NL solution
coda sslution
explain 10 poirs
frrm beseline
helper functions docstring
Intermediate variables and type
problem difficulty
problom tag
seff-roflection
Reasoning LLaMA 700 Models
Passg100
NL solution
code solution
explain 10 pairs
frorm baseline
helper functions docatring
Intermediate vartables and type
problem difficulty
problem tag self-retlcttin
Figure 17: Group by reasoning prompts averaged across all instruction prompts (top) for small
models and (bottom) for large models. For pass@1. @1 I, "explain 1O pairs" helps small models, and "helper function docstrings" helps large ones. The relative efficacy of each prompt converges to similar order for pass @ 100 for large and small models.
Combinations
GPT-4o
Reason, Instruct combinations:
& from baseline pass@k
seficefaction. adution
1.3
be docating,
sakitinn weak
par Auscciona docatinng instruction
100
Figure 18: Best combinations overall. We calculate the normalized pass @ k improvement with
respect to the baseline averaged across all 6 models for pass 1 (3.0 8B, 70B, 3.1 8B, 70B, 405B and
GPT-40) and 4 models (I.lama 3.0, 3.1 8B and 70B) for pass@ 10 and pass @ 100 on CodeContests test. We plot the top means and their corresponding prompt combinations for different sample sizes. 0 on the y-axis corresponds to the models' performance without COT.
D JUSTIFICATION FOR PROMPTING SPACE
D.1 REASONING PROMPTS NOT ADDITIVE
We describe methods that did not help enhance multi-turn CoT, specifically adding more complex exccution feedback and more steps of reasoning prompts. Our experiment result is shown in Table that before outputting the first code, stacking more reasoning steps hurt the performance, especially for Llama 3.1 70B.
21

---

Published as a conference paper at ICLR 2025
Llama
8B
Liama 3.1 70B
EASY
VERY HARD
EASY
VERY HARD
40
2.5
2.0
easseg
30
1.5
paxx-@k
passggk
40
8.1
30
10.
10a
10
10a 10e
101
10a
10a
10a
10
10
10t
10a
Best CoT
No CoT
Figure 19; We use the best CoT (i.e., reasoning and instruction prompt combination) found with Llama 3.0 8B and test directly with Llama 3.1 and Llama 3.1 70B on the easiest (easy) and the most difficult (very-hard) split of TACO.
Table 7: Stacking prompts hurt performance for Llama 3.1 70B. Each line in the table is added from the previous setup. +1 reasoning makes the model answer 2 reasoning prompts before code generation. +I instruction makes the model answer reasoning prompts and 2 instructions during code generation.
Llama 3.0 70B
Llama 3.1 70B
Number of prompts
1@3
100@ 300
1 I@3
100 300
reasoning instruction
11.2
40.0
24.5
59.2
reasoning
-0.4
-1.8
-2.0
-3.1
+ instruction
-0.1
+0.4
-4.0
-2.1
D.2 SIMPLE EXECUTION FEEDBACK SUFFICIENT
show Table S that execution feedbuck prompts with different granularity present low vari- with respect the pass rate, both in teighrtemprrturee setting (1.0, pass 100@300) and low- temperature setting (0.2, pass 1@3).
We posit that for challenging problems presented in the competitive programming benchmark, models generate wrong code not because the code is buggy by accident but because models do not understand how to solve the problem correctly. It highlights the fact that for competitive programming benchmark, algorithmic reasoning (to align what the models believe to be a correct solution with the ground-true solution), as elicited by CoT', impacts the performance more than bug-fixing ability (to align the emitted code with what the models believe to be a correct solution).
Table Execution feedback result on multi-turn CodeContests test set. Results are reported using 3-turn trajectories. We also include a single-turn repeated sampling for comparison. 1 @3 is estimated from 20 trajectories per problem under temperature 0.2. @00 is estimated from 200 trajectories problem under temperature 1.0.
Llama 3.1 70B
Ilama 3.1 8B
Feedback
Granularity
1@3
100@
1@3
100@
N/A (Single-Tur)
N/A
27.3
53.5
28.0
Binary
28.8
55.9
10.9
30.9
Failed tests (default)
29.5
56.2
10.9
29.5
Failed & passed tests
++
29.5
55.0
10.7
30.4
LDB (Zhong et al, 2024
+++
26.5
54.8
9,9
29.1
22

---

Published as a conference paper at ICLR 2025
E ABLATION STUDIES
E.I ABLATION OF RETRY PROMPT IN MULTI TURNS
the multi-turn setting, after giving the execution feeduaik we add at the end the user message a prompt to ask for another code solution. This prompt is fixed to "Give it another try" throughout the whole paper.
We conduct ablation experiment in which test failed (Analyze) and fix the public tests (Fixme), as well as their combination, after giving the execution feedback. The variants experiment with are:
explicit prompting reasoning about why the
Retry: "Give it another try." (Used in the paper)
Fixme: "Generate fixed version of the program to fix the failing test."
Analyze - Retry: "Analyze the execution feedback. If runtime exception, identify the source. If wrong answer, simulate and analyze how the input maps to the actual output in your code and where it differs from the expected output. After that, give it another try."
Analyze - Fixme: "Analyze the execution feedback. If runtime exception, identify the source. If wrong answer, simulate and analyze how the input maps to the actual output in your code and where it differs from the expected output. After that, generate fixed version of the program to fix the failing test."
Table 9: Ablation of retry prompt multi-turn CodeContests test set. Results reported using 3-turn trajectories without CoT prompting in 1@3/ 100@300 330. Both I@3 and 100@ 300 are estimated from 200 trajectories per problem under temperature 1.0.
Analyze Reetry
Analyze Fiime
Model
Retry
Fixme
Llama 3.1 8B
7.0 / 30.4
6.7 29.3
6.6 30.0
6.3 27.5
Llama 3.1 70B 24.1 56.2
25.2 / 55.7
25.2 54.6
24.9 55.9
We report the performance on CodeContests test set in Table 9 Our ablation shows that explicitly prompting the model to focus on the failing tests and fix it degrades the performance for Llama 3.1 8B in 1 @3 and 100@ 300. For Llama 3.1 70B, the 1@3 increases by 1.1% while the 100@300 drops. For Llama 3.1 70B, the ablation shows an exploration-explotatiiee exploitation trade-off between 1@3 and 100@300. We attribute the performance degradation in Llama 3.1 8B to the imperfect multi-turn ability.
E.2 ABLATION OF NORMALIZATION STEP IN SIMILARITY SCORE
We show in Figure 20 and Figure 21 the distribution and histogram of similarity score without the normalization step. The similarity score, therefore, measures the code generated by the LLM. Compared with 6 and & the fundamental trend does not change. The robustness against our normalization step shows that the LLMs we study are already able output coherent (in terms of variable naming and formatting) code within the same dialog.
E.3 ABLATION OF RFT DATA MIXTURE
As detailed in Appendix B.2 we collect 2 sets correct trajectories, single-turn (ST) and multi-turn (MT), from the problems CodeContests training set using Llama 3.1 70B. We perform LSH- based deduplication to a maximum of 50 solutions (in each set) per problem statement. We also decontaminate the sets from TACO test set as detailed in Appendix[]
We show the ablation of the following design choices:
Data Source: train on solutions generated by Llama 3.1 70B (RFT) or solutions in the CodeCon- tests training set (SFT).
23

---

Published as a conference paper at ICLR 2025
Llama3.1-80
Llama3.1-70B
2.5
2.0
2.0
1.5
Denaty
1.5
Density
1.0
1.0
0.5
0.2
0.4
0.6
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Similarity
Similarity
binary feedback
failed
passed feedback
feedback
Figure 20: Distribution consecutive code similarity scores (without the normalization step described when varying the execution feedback granularity.
in Appendix B.I
Within Correct
Trajectories
Within Incorrect Trajectories
2000
200
1500
Count
150
Count
2000
300
0.0
0.6
0.2
0.4
0.6
2.0
Similarity Scores
Similarity Scores
before multi-tum CoT RFT
after multi-turn CoT RFT
Figure 21: Histogram of the similarity scores (without the normalization step described in Ap- pendix B.I of consecutive codes generated by the model before/after multi-turn CoT RFT on CodeContests test set.
ST v.s. MT Trajectories: train on single-turn (ST) trajectories only, multi-turn (MT) trajectories only, or both of them (ST + MT).
Including CoT Response: train on code solutions and CoT responses or train on code only.
For SFT, we follow the training set cleaning process of Jain et al. (2024b). We conduct LSH-based deduplication the solutions in the training set limit maximum of 25 solutions per problem. We then construct a single-turn dialog with the user message being the problem statement and the model message being the code solution.
We use the same set of hyperparameters described in Appendix B.2 of all the ablation experiments. All the RFT experiments are finetuning for exactly epoch avoid over-fitting. For the SFT experiment, we finetune for and 2 epochs and report the best performance, which is at epoch.
We show in Table 10 the ablation result. We find that SFT hurts the performance compared the base model. We posit that it is because the SFT dataset is far from the model output distribution of Llama 3.1 70B. The reasons are:
1. Given that Llama 3.1 70B has already been heavily tuned in the solutions in CodeContests training set of less quality than the data presented in its post-training phase. For example, some imports in the Python codes are outdated (e.g., from fractions import gcd will throw ImportError since Python 3.9).
some code
2. The dialogs in the SFT set are constructed in mechanical way with only code body in the model response, therefore far from the dialog distribution, i.e., the interaction between user and assistant in natural way, that the Instruct series of Llama 3.1 has seen in the post-training phase.
This is similar to the finding by /etlur et al. (2024) that RFT more data efficient than SFT since the RFT dataset is closer the model output distribution.
24

---

Published as a conference paper at ICLR 2025
Our ablation shows that removing the CoT response will introduce slight performance drop. We also find that training on multi-turn (MT) data only provides better performance. We hypothesize that the single-turn (ST) trajectories solve the problems of which models already capable. Further reinforcement on these problems could potentially lead to overfitting and bias the model behavior towards trying to solve the problems in the first turn instead of enhancing its multi-turn capability.
Table 10: Ablation of RFT data mixture. We show the best performance of the ablation runs of the following choices: training single-turn (ST) multi-turn (MT) data, whether include the CoT response. We show the performance of Llama 3.1 70B without finetuning and finetuning on the given CodeContests training set (SFT) on the top as reference.
CodeContests / Test
Data Source
ST
MT
CoT Response
1@3
10@ 30
100@300
Llama 3.1 70B
X
24.1
43.8
56.2
CodeContests/train (SFT)
v
X
16.6
33.6
44.9
X
X
26.8
47.5
58.3
X
28.9
49.2
60.1
Llama 3.1 70B (RFT)
X
X
29.1
50.1
60.0
29.1
49.6
60.0
X
V
29.7
50.5
61.1
F BEHAVIORAL ANALYSIS
F.I RFT MODEL BEHAVIIR ANALYSIS
We show in Table IT the fraction of text characters by the total response length. We take into account the intermediary CoT response if CoT is used. RFT model significantly increases the text output around the code output, which could contain reasoning traces.
Table 11: Fraction of text characters (not extracted code) by the total response length. We also count the CoT response when CoT is enabled. The RFTed model outputs more text the response.
Non-Code Fraction
Model
Llama 3.1 70B
0.37
Multi-turn CoT
0.57
Llama 3.1 RFT
0.50
F.2 DOES MORE NON-CODE TOKENS CORRELATE TO BETTER PERFORMANCE?
We describe non-code tokens as responses to reasoning steps and natural language generated with code attempt. We look at the fraction corresponding to non-code tokens from all tokens for GPT--o and 3. 3.1 70B to understand their difference in pass rates across prompts. We made the hypothesis that more non-code tokens correlate with more reasoning and, therefore, overall performance, with the effect similar to the pause token (Goyal et al. 2024) or the thinking token (Herel Mikolov 2024).
However, as shown in Figure 22 we observe that the same reasoning prompt, as well as combinations with instruction prompt, leads to approximately the same number of tokens across models but different pass rates. This invalidates our original hypothesis. We believe the fine- tuning prompts post- test-training probably influence the most which prompts are effective with which model.
25

---

Published as a conference paper at ICLR 2025
GPT-4o
26
self-reflection
explain 1o pairs
24
problem tag
48
problem difficulty
22
NL solution
88 4004
solution
20
pass@1
helper functions docstring
EMPTY
18
8
use helper functions
16
check constraints
line
&
func. docstring
weak solution
12
algorithms
Llama 3.1 70B
24
88
22
20
pass@11
10
+
16
14
12
200
300
400
500
600
700
800
900
1000
Number of Reasoning Tokens
Figure 22: Comparison of average non-code fraction between GPT-4o and Llama 3.1 70B based on different prompting strategies. We sample from pool of 7 reasoning and instruction prompts (with index being instruction) commonly used in code generation, with prompts as presented in Appendix G
G PROMPTS
We list the prompts used throughout our experiments inspired by recent works in code generation (Zelikman al. 2023) Jain al.) 2024b Paul et al. 20244 Ridnik et al. 2024). We focus zero-shot prompting techniques specific to competitive programming problems or, more generally, to code generation. We classify prompts into two categories: reasoning and instruction. To determine this list, we ran experiments at small scale (pass @ 10) with over 30 prompts on 500 examples sampled from the CodeContest training set. We picked the most promising ones terms of final unit test pass and execution rates. Some of our prompts are adapted from recent works in competitive programming.
G. REASONING PROMPTS
Adapted from AlphaCodium Ridnik et (2024)
self-reflection:
Given the code contest problem, reflect the problem, and describe it in your own words, in bullet points. Pay attention to small details, nuances, notes and examples in the problem description.
26

---

Published as a conference paper at ICLR 2025
predict 10 pairs: Given the code contest problem and the provided examples, take the first examples and explain how its input leads to the corresponding output. Read carefully the problem description. Make the test explanations are consistent with them, and between themselves. The explanation must coherently and logically lead from the input to the output. Be succinct.
write code solution with guidelines: Your goal is come up with possible solutions to the code contest problem. Guidelines: Make sure each solution fully addresses the problem goals, constraints, examples, and notes. Each solution must have reasonable runtime and memory complexity less than three seconds on modern computer, given the problem constraints for large inputs. Double-check the solutions. Each possible solution must be able generalize to additional test cases, not just the ones provided in the problem description.
predict problem tag: Explain which two tags from the following list best apply to this problem: combinatorics, dynamic programming, math, bitmasks, number theory, brute force, data structures, divide and conquer, graphs, greedy, depth first search and similar, implementation, binary search, two pointers, strings, constructive algorithms, sortings, trees, disjoint set union.
predict problem difficuly: Given the code contest problem, your task is to evaluate the difficulty of the problem either easy, medium hard. Explain the difficulties of the problem and potential edge cases.
write natural language solution: Generate naive solution to this problem in natural language and then explain how you could improve it.
write helper function doestring: Explain which helper functions you will need to solve the code contest problem. Without implementing them, write their signature and a doc string explaining their purpose.
write intermediate variables and type: Explain what necessary intermediate variables you will need to solve the problem, specify their type and purpose in your solution.
G.2 INSTRUCTION PROMPTS
Adapted from AlphaCodium Ridnik et al. (2024)
use helper functions: Guidelines: You must divide the generated code into small sub-functions, with meaningful names and functionality. Variables names should also be meaningful.
double check the import, variable, constraints: Double-check the solution code. Make sure to include all the necessary module imports, properly initialize the variables, and address the problem constraints.
comment before each line: Write a comment before each line of code to explain your thought process.
docstring before each function:
Write a doc string before each function generated explaining its utility and expected inputs and outputs.
generate weak solution and second better one: Generate one solution in python, identify its weaknesses and then generate second better algorithm to solve the problem.
step by step: Think step by step and propose clever algorithm.
G.3 COT-RETRY PROMPTS
For given problem, the model first solves it without any reasoning. If the first code generation is incorrect, add an instruction prompt, "write code solution with guidelines," to separate problem- solving and code writing. If the second code generation is false, we add another instruction prompt, "generate weak solution." These prompts are combined with the default (failed tests) execution feedback after each turn. This setup provides up to +6% improvements for the Llama 3.1 series.
We show example extracted from CodeContests generated by Llama 3.1 70B the following page. We picked this prompt combination based on the results in Figure 18 The instruction "weak solution" appears most often as part of the top best mean pass @ improvement from the baseline across models for CodeContests. As the reasoning prompt varied more across sample sizes, we tried all combinations and empirically "write code solution with guidelines" worked best in CoT-retry for CodeContests and TACO.
27

---

Published as a conference paper at ICLR 2025
User (Turn 1)
Here competitive progranning question: The generation external menory contains array of integers a[1 n] [a_1, a.2, .... a_n].
This type of ieeooy does not support chenging the value an arbitrary element. Instead, it allows you rotate) it by any offset and insert it back into the sane place..
out segment of the
array, cyclically shift
Technically, each eyclic
consists 1. You may select arbitrary indices and sigrrs segment.
consecutive
the boundaries af
2. Then you replace the segment a[l...r] with it's cyclic shift the left by arbitrary offset d. The concept of cyclic shift can be also explained by following relations:
cyclic shift of the sequence [3. 4, the left by the offset and the sequence 14, J. 11 is eyclic shift of the by offset
[1,
sequence [3, 1,
For exanple, if a [1, (3, 8), 51, then choosing =
4 and = yields 2 to which then takes the place of of the original
segnent af2 ... 4] [3, 2, 8]. This segnent is then shifted by the offset
left, and elements of the segnent. In the end you get a = [l, (8, 3, 2), 5J. Sort the given array using
[8.
than cyclic shifts of any of its sagnents. Note that you don't need to ninimize the number of cyclic shifts. Any nethad that requires or less cyclic Input
accepted.
The first line contains an integer (1 (155 t 1880) the number of test cases.
The 2t lines contain the descriptions of the test
The first line of each test case description contains on integer n (2 S n & 50) the length of the array. The sacond line consists of sppca-separated elements of the array a_i (-18-9 I a_i S 18*9). Elements of array may repeat and don 't have to be unique. Output
Print answers
all input test cases.
The first line of the answer of each test case should contain an integer k (8 S k 5 n) sert the array. The next k lines shauld contain the artians formatted as "I d' (without quates) where and (1 $ segnent being shifted, while d
the number of actions descriptions
n) the boundaries of the offset value. Please remember that only the cyclic shifts to the left are considered
1) (
chosen segnent will be Note that you not required to find the mini nun number of cyclic shifts needed for sorting. Any serting method where the number of shifts does not exceed n will be accepted.
by the offset to
to the left.
If the given array a is already sorted, one of the possible answers is k = and an possible answers, you may print
empty sequence of cyclic shifts. there of them.
Example
Input
2
3
1 2
1
Output
2
1
3
2
2 4
2 3
5
Nate
Explanation of the fourth data set in the example:
3] + [2, {4. 5, 1). 3] selected and shifted left by (5.
The segment
segment
...
selected shifted left by D(2,
5, 1, 3)] [(1, 3, 2, 4, 5)]
28

---

Published as a conference paper at ICLR 2025
After that the segnent all 2] is selected and is shifted the left by 1:
IO,, 3), 2, 4, 5] + I(3, 1), 2, 4, 5]
And in the and the segment a[1 ...
selected and shifted the left 1: your self-reflection
[C3, 1, 22, 4, 5] + [(I, 2, 3), 4, 5]
problen, your goal write valid Python code with stdio that correctly solves the problen. Your code should be enclosed triple backticks like so: ""python YOUR CODE HERE Use the backticks for your code only.
contest problen
Assistant (Turn Response)
python
def port_aray operations
arr sart() for if arr[i] 1= arr[i-1]:
(anga(a))())r));
operations. append((i, 1+1, i (err.index(arriii (1-1))))
(erratioos..apen(((...
for operation in operations:
(operatinn)))) 1, operation[1]] operation[2]]
main(:: for
range (Int((iuutt)
n intpin ())
list
ser (ort_array(arr)
if name__ main O
== m__nni__".
User (Tur 2)
Your code failed the following tests:
input "4\n2An2 ANn3Ant 2 1An4/n2 4 3\m5\n2 5 4 failed:
Expected output "1Amt 2 1na1nn2 1Aa2Ant 3 2\ 1Na3Ant 3 2\m3 2An4 5 1" but got '25n1 -1in2 8in2/n1 -1Xm3 IN44Nn1 -1Nn2 n3 3 1And 4 2\n5 5
1A44 2\n5Xm1 -1in2
Give another
Your code should be enclased in triple backticks like so: python YOUR CODE HERE Use the backticks for your code anly, goal is to with possible salutions to the code contest problen.
Guidelines:
Make sure each solution fully addresses the problem goals, constraints, examples, and notes.
Each solution must reasonsble runtime nenory complexity less than three seconds on a modern computer, given the problen constraints for large inputs. couble-chect the solutions. Each possible solution be able to ganeralize additional test cases, not just the ones provided in the problen description.
Assistant (Tum Response)
pyyhhh
sort_array ( operations 8 while (arr):
if i and arr[il <= arr[i-1];
shif
operations)
i += print (len (Ien(oparttti)) for operation in operations:
(rint(opprationn0@)11 operation[l]+l, (peration(2])
def shift_arr (arr, index, operations):
segnent arr[index+1:] segment_len len (segnent)
29

---

Published as a conference paper at ICLR 2025
max (eegmeeettt)
max max_val_indes segnent. index (ndexxaax_val)
shift nax_val_index for 1 in range (1ndex*11 index=segnent__en)
segment[i-index-11
segment[i]:
shift -=
arr arr[index=1:] segnent[=shift:] segnent[:-shift]
operations.. append index seggensegnnn,,nn shift))
def main(:
for in range ())
int(input()) (t)
(ist(map(in,, ()put(..ppit(())
sort_array(err)
if __name__ = "__main__.; m)in()
This solution sorts the to the front, creating new segment that starts at the current index 'i'. The number of shifts required shifts are stored in the operations" list. The process is repeated until the entire array
shifting the naximum elenent of each unsorted segnent
the maximun elenent the frant is calculated, and these
User (Tumn 3)
code failed following
input '4\m2An2 1Nn3Anl
2\m5\n2 failed:
Traceback (most recent call last):
File "python_iopsirs.. by" line 126, in main (eecute_code(coopieeee))
File "bythoo_iopairs.pp"" line 40, in execute_code
exec(code, ("__name_.. =: "__main__""
File "<source>", line 32, in <nodule>
main(
File "Ssource>", line 29, in main (art))
File "<source>", line 7, sort_array shift_arr(arr, 1-1, operations)
File
20. shift_arr
if segnent[i-index-1 -1] segment[i]: range
IndexError: list
Give another
Your code should be enclosed in triple backticks like so: "*"python YOUR CODE HERE backticks your cede only. Generate solution this problem make sure you pass all the tests then explain how you could improve it.
Assistant (Turn 3 Response)
.** pptthnn
def sor (arr) operations len(arr)
for range(n):
max max_index =
range n):
if arr[j] < anddnxx_index] max_index
if max_index 1= nax_val arr[nax_index]
del [rr_nex_iddex] arr. insert(i. max_val) Ieeraaiios..apeeddee
max_index=1, nax_index-
print (Ie
for operation in operations:
(pperatiooa operation[1], operation[2])
main
for in range (int (input ())):
30

---

Published as a conference paper at ICLR 2025
(ntputtt
arr list (int, input input().sllit()))
sort_array
n_me-__ "__main___"
m)in()
This naive solution simply finds the maximun element each segment of the array and shifts the front. This is in way that correct position in single pass, avoiding the need for repeated passes.
element moved its
However, this solution has time complexity of 0(n*2) due the nested loops used find the maximum elenent in each segnent.
To imprave this solution, we can use a more efficient sarting algorithn, such as nerge heap sort, which sorting algorithn that takes advantage of the fact that the array is being sorted in- place, as
complexity of log n) We also
sart selection sort.
Alternatively, we can use different approach selve the problem, such as using data structure that allows for efficient insertion delation of elenents, using more advanced algorithn that can sort the array in less than n passes.
Here is improved solution using bubble sart:
thon
def sort sort_array(arr)
operations []
n = len(arr) range
far in range(n-1)) if arrtil arr[j+1]:
arr[il. arr[j+]] arr[j+1], arr[j]
operations... (perations.append(())))) appen j+2, 1))
print (1ettoprataiooo))))
for operation in operations:
(oprratpptt.) operation[1], operation[2])
main(:
for range ())::
n int (input()) (nap((nn,
sort_array(arr)
if name__ == main____ m)in()
This improved solution still efficient than the naive solution due to the fact that it only requires single pass through array.
complexity 0(n*2), slightly
H EXAMPLES WRONG CoT
Some problems are solvable by the model without CoT the single-turn setting but become wrong when we add specific reasoning and instruction prompts. Some instruction prompts can lead to incorrect planning- The weak solution prompt, although effective when done correctly as shown by its high performance across models, can bias the model towards brute force approach that is unable to recover from as shown below Example and Example 2. In Example 3, the reasoning prompt for the model is to define intermediate variables which will be used in the code, while the generated code does not contain the intermediate variables described earlier.
These examples generated with Llama 3.1 8B and extracted from CodeContests.
Example 1: Brute Force Attempt
pythan def sa,ve c): combinations [(i, i, k) for i in range(a + 1) for j in range 1) for k in range(c 1)]
31

---

Published as a conference paper at ICLR 2025
min_diff Float('inf')
for al, b1, ct in combinations: az, b2, if a2 >= 0 and b2 == and c2 >= 8: abs[(al
b1,
cl) (a2 c2))
nin_diff min(min_diff, diff)
return min_diff
int int ()
in ;t)
a, b. c map(int, print(salveea, b, c))
This naive solution tries generate all combinations of one-minute songs, two-minute songs, and three-minute songs that add up to the total nunber of each song type, and then calculates absolute difference ninimun difference is then returned.
for each combination. The
However, this naive solution has time complexity of O(n*3). where the total number of each song type. This is because there are 0(n) conbinations for each sang type, and need try all conbinations. This solution may not be efficient for large inputs.
To improve the solution, we can use more efficient algerithn to generate conbinations example, we can divide set of objects into two subsets such that the of objects in each subset is equal.
concept partitioning,
more
way
Ne can also use dynanic programming to store the internediate results and avoid redundant calculations.
However, problen constraints are large, naive sufficient all tests within reasonable time.
Example 2: Same Code Twice
"haythhn import sys
def solve
int(innut:()
list(map(int,
max_k
for i (38): count = sun((x >> i) & far x in a)
max__ax_ count)
possible_k set()
for k (1, max_k 1): bed.bllkek..dddck (rot(tssorto((pessstte...)
int(inntt O)
in solve()
**Meaknesses of the first solution:*s
1. The solution uses brute- approach to count the number of elenents with a at complexity 0(n m), where the length of
each bit pasition, which has the array and a is the number of bit positions. solution
possible values complexity of ((n) for adding and removing elenents.
which time
3. The solution uses single print statement to print all the possible values which time complexity of O(n).
seSoond solution:**
python
import sys
def solve():
(inputt))))
a list
for 1 in range (30): sun((x
max_k nax (nax_k, caunt)
32

---

Published as a conference paper at ICLR 2025
possible_k set()
for k in range(1, max_k 1): possible_k, aaasible_k.add(// (dd
rrinttssortedppossble...
def main(::
= int(input
for range (t): salve()
1f nane...... == main__" main()
InInooveeents::
The second solution the algorithn the first solution, but it the main logic into separate function "solve() ()* and the input /utpuut logic into a separate function "main(". This
code modular and 2. The second solution uses the same data structure (set) to store the possible values mare
understand.
k,
each bit position. it uses single pessible the
algarithn find
k.
Instead of iterating find the maximum passible value of k.
The second solution of k, but it uses more efficient algorithn to print the values in increasing order. Instead using single print statement, it separately.
the print statenent print all the possible values
loop print value
complexity:
The tine complexity of the first solution 0(n As where the length of the array and n is the number of bit positians.
The tine complexity of the second solution is 0(n where the length of the array and n is the number of bit positions. This is because the solution uses single possible
pass over print each value separately. s*Space complexity:*4
array
k, then
loop
The space complexity of both solutions (mm), where the nunber bit positions. This is because the solutions use set to stare the passible values k, and the size
proportional the number of bit positions.
Example 3: Not Using Intermediate Variables Defined Before
Internediate Variables
"k*sw ("int"):
array range
possible values for each element, respectively.
"***"** ""eettorint>"); The input array with indicating unknown values. **'c'* ('int"): : The nunber of elements "a" that -I after replacing all -1s with randon values.
"n"ino" ('int"): The for each increment in "c'.
(which is equivalent the reciprocal
an" expeccedcosst elenents in "a" be the same, nodulo "10'9 7"
("long l)ng"): The expected number operations get
tempSun ('leng long"): A temporary variable to held the sun of expected costs for each possible configuration of "a" after replacing -1s.
** nunnCnffiis *** ("long long"): : The number of possible configurations of "a" after replacing
sA-result ("long long ) The result, which is the expected cast of "a" nodula 110"9 7"
=o"nod"** const long long") The modulo value (1109 7*).
.. ''pyhhan
import sys
Input sys. ses.stdin.. readline
inv(a, p): res = p-2
while p:
1f p & 1: res = (res a) x MOD
return res
def C(a, b): if ==
if =s b: return 1 return (((fac[aJ*i fac[a-b]ei f)c(b (fac[b]. HOD)
33

---

Published as a conference paper at ICLR 2025
(int, input ()put().splle split()) ())litt
(nap impue O split
def dfs(x):
global if == len(a) 1:
C(k-1, a[x])
C(k-1, a[x]+1)
ans (ans v142)8H0D
(ans NOD x2)
return
= C(k-1, a[x]]
v2 c(k-1, a[x]+1)
C(k-1, a[x]-1)
ans (ans v1 v2 v3)XN0D
:
a[x+1] =
(x+(x
ans (ans 1)8MCD
MOD 18*a9 == -1
[1]+108001
for in range (1, 100881):
fac[j] (fac[j-1]<i)8N)
(1 RM
(0)
print ((ansSMOD MDO) HHDD)
I DE-CONTAMINATION BETWEEN CODECONTESTS AND TACO
We found that there non-zero overlapping between CodeContests training set and TACO test set. Therefore, after gathering successful trajectories from Llama 3.1 70B on CodeContests training set, we further conduct de-contamination to filter solutions the problems that overlap with problems in TACO test set. We mined the contaminated problems as follows.
We note that exact string matching will result a lot of contamination remaining undetected due the different latex parsing and format between benchmarks. We, therefore, use an off-the-shelf sentence embedding model to compute sentence similarity between problem statements from CodeContests training set and TACO test set. For each problem Puee in TACO test set, we set the threshold of sentence similarity to 0.8 to obtain similar CodeContests problems (P (PCodeContstss ). We take the first
solutions from Piaco and run each solution against all the unit tests available of each similar problem PeodoContests If any of the solutions passes the unit tests, we label this as a contamination.
Our dataset mined from the Llama 3.1 70B output CodeContests comprises solutions 7238 problems in the training set. We detect 288 problems contaminated with the TACO test set, resulting in solutions to 6950 problems after filtering. This process further removes, after the -asee de-duplication, a total of 6422 entries from the single-turn trajectories and 7463 entries from the multi-turn trajectories.
We show an example of a contaminated problem in CodeContests training set and TACO test set below.
Contaminated CodeContests Training Set Problem
array with length n, you perforn operations. Each operation like this: choase two adjacent elenents from ged(x, y), where ged denotes the [greatest connon divisor1(https: //en. wikipedia.org/ ) divisor).
and y, and replace one of then with
What is the nininum number of operations you need make all of the elenents equal 12
Input
34

---

Published as a conference paper at ICLR 2025
The first line of the input contains integer 2800) the number of elenents in the array.
The second line contains n space separated integers al, a2, an (1 S a1 189) elements
array.
Output
Print -1, impossible to turn all numbers to 1. 0therwise, print the minimun number of operations needed make all nunbers equal to
Examples
Input
2 2 2
Out
4
Output
-1
Input
3
Output
In the first sample you can turn all numbers to using the following moves:
[2,
2,
3, 4,
61.
[2.
1. 3.
1. 6]
[2,
[1,
1, 1, 67
[1,
1. 1]
Me can prove that in this case it is not possible to make all numbers ane using less noves.
Contaminated TACO Test Set Problem
You have an array with length n. you can perforn operations. Each operation is like a, say y, replace of gx, y), where ged denotes the greatest comman divisar.
this: choose adjacent elenents
What the nininum number of operations you need make all of the elenents equal 1?
Input
The first line of the input contains one integer (1 2088) the number of elenents in the array.
35

---

Published as a conference paper at ICLR 2025
The second 1ine contains separated integers al_l, a2, aÑ (1 $a_(1)$ S $18*95) the elements of the array.
-utputt
Print -I, it inpossible to curn all numbers to 1. Otherwise, print the minimun number of operations needed make all nunbers equal to 1.
-..--Exa Examples
Input
2 2 Output
5 Input
4
Output
-1 Input
3
Output
In the first sample you can turn all numbers to 1 using the following 5 moves:
[2, 2, 3, 4, 61. 12, 1, 3, 4, 63 12, 1, 3, 1, 6] [2, 1, 1, 1, 6] [1. 1, 1, 1, 6]
D1 1, 1,
Ne can prove that in this case it is not possible to make all nunbers ane using less than
J CONTAMINATION OF TACO TRAINING SET AND TEST SET
We also find that there are non-zero overlaps between TACO training set and test set. These overlaps, despite having different URL, have near identical problem statement. We find that this could be attributed to the fact that on the Codeforces platform, harder problems from easy contest (div2) could appear also in harder contest (divl) easier problems. We show example below, in which in http:///ddfoorcccccccoooooooolss training set the problem URL is htt.:///w.e...e///oo///////////a 840/A test it httpss:/ccoefforress
e44// and in
Contaminated TACO Training Set Problem
Leha like all kinds of strange things. Recently he liked the function F(n, k). Cansider passible k-elenent subsets
2, .... n]. subset mininal element in it. F(n, k) nathenatical expectation of the minimal element anong all k- element subsets.
But only function does not interest hin. He wants do interesting things with it. Ham brought hin arrays and each consists of integers. For all J such that i. S the condition Ai 2 Bj holds. Help Leha rearrange the nunbers in the arrey already rearranged array.
the <inage> maxinally possible,
Input
First line of input data contains single integer (1 S S 2-105) length of arrays and
Next line contains integers al, a2, $ ai S 189) array A.
Next line contains n integers bl, 62, bn S bi S 189) array B.
Dutput
Output integers a'1, a'?, a'n array A which is permutation of the array A.
36

---

Published as a conference paper at ICLR 2025
Examples
Input
7
3
Input
4 6
8
8
2
putput
Contaminated TACO Test Set Problem
Leha like all kinds of strange things. Recently he liked the function F(n, k). Consider all possible k-element subsets of the [1, .... n]. subset find mininal element in it. F(n, k) mathematical expectation of the minimal element anong all k-
But only function does not interest hin. He wants do interesting things with it. Ham brought hin arrays and B, each consists of integers. For all such that 1, S m the condition {_(i) 2 B_(3) holds. Help Leha rearrange the numbers in the array that possible, where A' is already rearranged array.
$1sun_(1 1)*(m) (1)"(Nin)) B_(1))s naximally
-np--Input--
First line of input data contains single integer (1 & & 2-10-5) length of arrays
integers a_1, a_2, a_(m) $ a_(i) 10*9) array A.
Next line contains n integers b_1, b_2, .... (n(m) (1 s (_ 18-9) array B.
Output integers a'1, a'_2, .... a'_(m) array permutation array
Examples
Input
5 3
21323
Output 4 7 5
Input 6
8 2
2
Output
37

---

Published as a conference paper at ICLR 2025
K UPPER BOUND PERFORMANCE ESTIMATION
Throughout this paper, we regard the CodeContests test set as a black box and use the performance of the whole benchmark as the signal for analyzing different reasoning, instruction, and execution feedback. However, optimizing the performance of these prompl variants on per-problem level will further boost performance. In this section, we aim provide an upper bound estimation if we select the CoT prompt based on the oracle, i.e., the best test set performance of each problem in the set of prompts. We do not intend the number presented in this section be compared with the existing methods presented in the main text, as the performance of the test set is exposed, but rather to provide an estimation of the potential room for improvement.
K.1 ADAPTIVE COT PROMPT SELECTION
Based on our grid search of 63 reasoning instruction prompts, presented in Appendix C.1 and summarized in T We post-hoc select the reasoning and instruction prompts, which induce the highest performance per problem rather than over the whole dataset. Table presents the potential room for single-turn performance improvement on CodeContests test set. The best combination per problem is selected based on the best performance in terms of pass @ 100, and the pass@ is reported using the same prompts selected by pass @ 100.
Table 12: Upper bound adaptive prompts on CodeContests test set chosen post-hoc from the 63 prompt single-turn CoT grid search (200 samples per problems generated with temperature 1.0). combination refers to reasoning x instruction prompt. The results for the best combination per dataset are the same as the ones presented in Table 3
Best combination per dataset
Best combination per problem
pass @
pass @ 100
pass @
pass @ 100
Llama 3.0 8B
1.5
17.3
2.5
22.6
Llama 3.0 70B
5.3
33.1
8.3
42.4
Llama 3.1 8B
4.0
26.1
5.3
41.5
Llama 3.1 70B
16.1
54.1
18.3
63.1
K.2 ADAPTIVE EXECUTION FEEDBACK GRANULARITY SELECTION
We show in Table 13 the post-hoc selection of execution feedback granularity based Table X to estimate the upper bound if we select the best granularity per problem in the multi-turn setting. Since in Table 8 @ is estimated from 20 trajectories generated with temperature 0.2 and 100@300 is estimated from 200 trajectories generated with temperature 1.0, we report the upper bound by selecting the best execution feedback granularity separately in both setting.
Table 13: Upper bound adaptive execution feedback (EF) on CodeContests test set chosen post-hoc
from the 4 execution feedback granularity: binary, failed tests, failed & passed tests, LDB, The number for the best dataset EF extracted from Table All experiments are in the multi-turn setup with maximum of 3 turns.
Best dataset EF
Best problem EF
1@3
100@300
1@3
100@300
Llama 3.1 8B
10.9
30.9
13.1
34.8
Llama 3.1 70B
29.5
56.2
33.6
58.2
38


Reviews:
[{"review_id": "WNp5LYEURV", "reviewer": "Reviewer_HALe", "summary": "The paper investigates the effects of varied prompting techniques, including chain-of-thought (CoT), reasoning, and execution feedback prompts, on the performance of large language models (LLMs) in multi-turn code generation. It focuses on understanding the mechanics and efficacy of these prompts on complex programming tasks using benchmarks like CodeContests and TACO across models of different sizes (e.g., Llama 3.0/3.1, GPT-4o). Unlike simpler code generation tasks, competitive coding demands algorithmic reasoning and has not been widely explored with unified evaluation metrics for multi-turn code generation. \n\nKey contributions include:\n\n1. **Framework for Multi-Turn Code Generation**: The paper establishes a unified framework that enables a mix of single-turn and multi-turn prompting techniques, incorporating reasoning, instruction, and feedback prompts. This approach aims to build robust baselines for multi-turn performance and to measure trade-offs between sampling budgets and model capabilities.\n\n2. **Experimental Insights**: The authors’ extensive grid search reveals that combining reasoning and instruction prompts significantly enhances single-turn model performance, especially for larger models and more difficult tasks. However, for multi-turn settings, while combining with CoT prompts boosts performance, using multi-turn alone has limited benefits and can even reduce performance at higher sample budgets due to less diverse outputs.\n\n3. **Internalized CoT**: The study adapts the RFT, which allows models to internalize multi-turn CoT reasoning during training. RFT-trained models demonstrate improved performance and reasoning abilities in the absence of CoT prompts at inference, showing promise for scaling up multi-turn code generation capabilities.\n\n4. **Exploration-Exploitation Dynamics**: The analysis indicates that detailed feedback can induce exploitative behavior in models, which is beneficial for simpler problems but counterproductive for complex ones where diversity in code generation is critical. This highlights the need for feedback strategies that balance between exploiting known solutions and exploring new ones, particularly in high-difficulty settings.\n\nIn conclusion, this work underscores that multi-turn code generation is not inherently beneficial without adaptive strategies like CoT prompts and RFT. The findings advocate for tailored prompting configurations based on problem difficulty and suggest that further advancements in multi-turn code generation could benefit real-world applications such as repository-level LLM agents capable of reasoning, self-repair, and complex interaction with environments.", "questions": "1. Diversity in Rejection Sampling Fine-Tuning: The paper fine-tunes the model on correctly generated trajectories. Given the potential for this approach to bias the model towards successful solutions, would the authors consider including near-correct or diverse trajectories to introduce a broader range of reasoning paths? This could enhance the model’s adaptability to different problem-solving strategies.\n2. Adaptive Prompt Selection for Reducing Computational Requirements: The extensive grid search across prompt types and model configurations is resource-intensive. Have the authors explored adaptive or meta-learning techniques to streamline prompt selection, reducing computational demands while still achieving optimal prompt configurations? Such techniques could make the framework more accessible for settings with limited computational resources.\n3. Detailed Analysis of Prompt Failure Modes: The paper mentions that certain CoT prompt configurations may degrade performance, particularly in multi-turn settings with higher sample budgets. Could the authors provide a more granular analysis of these failure modes? Understanding specific prompt combinations or scenarios where performance degrades could guide practitioners in avoiding ineffective configurations.\n4. Balancing Exploration and Exploitation in Feedback Prompts: The study notes that finer-grained feedback prompts may encourage exploitative behavior in models, potentially limiting performance on complex tasks. Have the authors considered any strategies for adjusting the feedback granularity dynamically based on task difficulty or performance? This could help achieve a balance between exploration and exploitation for more challenging problems.\n5. Transferability of the Findings to Other LLM Architectures: The paper experiments primarily with the Llama series and GPT-4o models. To what extent do the authors believe that the findings, especially related to prompt configurations and RFT, would transfer to other LLM architectures with different pretraining characteristics? For instance, would models with fewer parameters or more domain-specific training exhibit similar responses to the same prompting strategies?\n6. Future Directions for Enhancing Multi-Turn Reasoning in LLMs: Given the current results, what specific directions do the authors see as most promising for advancing multi-turn reasoning capabilities? Are there any particular technical approaches, benchmarks, or modifications to current LLM architectures that the authors believe could improve multi-turn performance on reasoning-heavy tasks?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**Originality**:  \nThis work provides a unique contribution by systematically exploring multi-turn code generation for large language models (LLMs) through diverse prompt configurations, including **chain-of-thought (CoT)**, **reasoning prompts**, and **execution feedback**. While prior work has explored CoT and single-turn code generation, this paper’s emphasis on **multi-turn prompting in competitive programming benchmarks** addresses an under-explored area. The study’s integration of multi-turn CoT with finely tuned execution feedback as a **comprehensive prompting framework** offers an original approach to improve model reasoning in multi-turn contexts. Additionally, the use of **Rejection Sampling Fine-tuning (RFT)** to internalize CoT reasoning across turns is a creative and impactful extension, enabling models to generalize reasoning capabilities without prompting at inference time.\n\n**Quality**:  \nThe paper demonstrates high quality in its rigorous experimental design and breadth of evaluations. Through an extensive **grid search** of prompt combinations, the authors provide a robust dataset for comparing single-turn and multi-turn settings across a **variety of model architectures and sizes** (Llama 3.0/3.1, GPT-4o). The benchmarks, CodeContests and TACO, are well-chosen for evaluating competitive coding capabilities, allowing the paper to contribute valuable insights for LLM applications in algorithmic and complex reasoning tasks. Methodologically, the work includes detailed ablations, sensitivity analyses, and cross-model comparisons, ensuring that the findings are well-supported and generalizable.\n\n**Clarity**:\nThe paper is well-structured, clearly outlining the motivation, objectives, and experimental setup. The **framework for prompt deconstruction** is presented systematically, with visual aids like **Figure 1** that clarify the operational structure of single- and multi-turn code generation. Each type of prompt (reasoning, instruction, feedback) is explained with appropriate examples, making the framework comprehensible for readers. The authors’ discussion on **exploration-exploitation dynamics** and performance limitations in multi-turn settings enhances clarity by contextualizing how feedback granularity impacts model behavior. Additionally, the conclusions are tied back to the primary research questions, reinforcing the relevance of the findings.\n\n**Significance**:  \nThe study makes significant strides in understanding and advancing multi-turn reasoning for code generation in LLMs. By introducing a unified prompting framework and evaluating its efficacy in algorithmically demanding benchmarks, the paper sets a strong foundation for future research in **multi-turn, reasoning-heavy applications** for LLMs. The RFT approach has implications beyond competitive programming, as it suggests that **prompted reasoning behaviors can be internalized** within the model, potentially reducing computational costs for other complex, multi-step tasks. Given the trend toward **LLM-based agents** capable of autonomous decision-making, this paper’s insights into adaptive CoT, feedback prompting, and exploration-exploitation trade-offs are timely and have high applicability.\n\n**Conclusion**:\nIn summary, this paper’s originality in multi-turn prompting, its methodological rigor, clarity, and significance make it a valuable contribution to the field of LLM code generation. The findings not only enhance understanding of CoT in multi-turn settings but also provide practical strategies for optimizing reasoning in LLMs, making this work highly relevant for both researchers and practitioners.", "weaknesses": "While the paper makes significant contributions to the understanding of multi-turn code generation in LLMs, there are a few areas where improvements could be made to strengthen the study further.\n\n**1. Limited Scope of Multi-Turn Framework**  \nThe paper’s framework, while well-designed for competitive programming benchmarks, focuses primarily on **chain-style multi-turn code generation**. This approach limits the exploration of more complex **tree-structured trajectories** or **backtracking mechanisms** that could better emulate real-world multi-turn interactions in agent-based LLM applications. Exploring additional trajectory structures—such as decision-tree-style paths where LLMs could backtrack or revise earlier steps—would add depth to the study and provide insights into more adaptive reasoning scenarios. Extending the framework to incorporate these structures could lead to a more comprehensive understanding of multi-turn reasoning dynamics.\n\n**2. Potential Bias in Rejection Sampling Fine-Tuning (RFT)**  \nAlthough Rejection Sampling Fine-Tuning (RFT) is a compelling method for embedding reasoning behavior, the paper’s reliance on **correctly generated trajectories** may introduce a bias toward successful patterns, potentially limiting the model’s adaptability to a broader range of problem-solving approaches. By focusing only on “successful” reasoning patterns, the model may overfit to specific prompt types or problem formats. Future work could consider **diversifying the fine-tuning data** to include not only correct but also nearly correct attempts with varied prompts, allowing models to learn from a broader array of problem-solving paths.\n\n**3. Dependence on High-Resource Settings**  \nThe paper’s approach requires considerable **computational resources** due to the grid search over a large set of prompt combinations and the use of multi-turn, CoT, and feedback prompts across different model sizes. This may limit the practical application of the findings for research groups or practitioners with restricted compute budgets. Introducing methods for **adaptive prompt selection** that can identify effective prompt configurations more efficiently (e.g., meta-learning approaches) could make the approach more accessible and scalable, especially for real-world applications where computational resources may be constrained.\n\n**4. Limited Discussion of Prompt Failure Modes**  \nThe paper could benefit from a more detailed **analysis of prompt failure modes**, particularly in cases where certain prompt combinations degrade performance in multi-turn settings. While the paper briefly notes that CoT may harm performance by obstructing LLM context or encouraging suboptimal plans, it lacks a thorough exploration of specific cases where multi-turn CoT and feedback combinations lead to failure. A more in-depth examination of these failure patterns, especially for challenging tasks, would provide actionable insights into optimizing prompt configurations and mitigating these issues in future iterations.\n\n**Conclusion**  \nIn summary, while the paper delivers strong contributions to multi-turn prompting strategies, expanding the framework’s scope to include more diverse trajectory structures, and refining RFT to include a wider range of training data. Additionally, addressing the high computational costs and analyzing prompt failure modes more deeply would make the work more robust and accessible for practical application. These improvements could further strengthen the study and support its stated goal of advancing multi-turn reasoning capabilities in LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4twVrUx4mj", "reviewer": "Reviewer_NFbK", "summary": "This paper evaluates various prompting strategies in single-turn and multi-turn code generation settings. For single-turn generation, it explores chain-of-thought (CoT) prompting, dividing it into reasoning prompts and instruction prompts. In multi-turn generation, the paper investigates multiple error feedback strategies. The evaluation covers the CodeContests and TACO benchmarks, focusing on Llama models and GPT-4o. The authors also fine-tune the models on multi-turn CoT data to internalize reasoning within the LLMs.", "questions": "Please answer the questions in the weakness section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "-\tThe paper brings together the evaluation of a wide range of prompting strategies in both single turn and multi-turn for multiple models on a challenging code dataset.\n-\tExtensive set of experiments with an extensive grid search\n-\tInteresting idea of splitting the CoT prompts into reasoning, instruction, and feedback so that we can understand what strategies work best for different model sizes and problem difficulties. \n-\tMulti-turn code generation findings reinforce insights from previous papers --- just multi-turn is not sufficient but combining with CoT might improve performance. \n-\tFine-tuning on multi-turn CoT data is a novel approach that shows promising results.", "weaknesses": "- **Limited novelty**: There is substantial existing research on CoT and self-repair in the code domain, which somewhat limits the paper's originality.\n\n- COT-Retry prompts seem like an arbitrary combination of prompts. Why this particular combination?\nThe paper lacks any chain of thought prompting specific to reasoning about why the test failed or how to fix the test.\n\n\n- For additive reasoning prompts experiment in table 6, what happens if each sample uses a different COT prompt? \n\n\nSome of the details are missing/not clear that gives me less confidence on the results:\n\n1.\t**Baseline for Fine-Tuning**: There should be a baseline that fine-tunes on problem-solution pairs (without CoT data). This would help clarify whether RFT fine-tuning offers benefits beyond simple fine-tuning.\n\n2.\t**Clarity on CoT and Multi-Turn CoT**: Tables 1 and 2 could more clearly explain CoT and multi-turn CoT strategies. Are they different across models? Are they also different across sample sizes?\n\n3.\t**Another baseline for table 8**: How does table 8 results compare to just resampling from the base model? Is feedback even playing a role in these multi-turn settings? \n\n4.\t**Explain Figure 6**: Why does the multi-turn CoT differ from the base model at max_turns=1? Shouldn’t they be identical if CoT-Retry doesn’t apply CoT in the first turn?\n\n5.\t**Misinterpretation of results**: Some results are misinterpreted in the text. For instance, Figure 4 uses different axes across charts, making it difficult to interpret trends accurately. . Also, we cannot  say that 1% to 2% is better than 30% to 40% or other way around.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dGsjmL2NiD", "reviewer": "Reviewer_8tPn", "summary": "This paper investigates the effects of various prompting strategies on large language models (LLMs) for code generation tasks, focusing on multi-turn interactions and computational requirements. The authors conduct a comprehensive study using competitive programming benchmarks (CodeContests and TACO) across multiple LLM families and sizes. They explore single-turn vs. multi-turn code generation, chain-of-thought (CoT) prompting techniques, execution feedback in multi-turn settings, and the impact of model size and problem difficulty on prompting effectiveness. The study also examines fine-tuning models on multi-turn CoT data.", "questions": "1.How well do the findings generalize to other types of code generation tasks beyond competitive programming? Are there more complicated benchmarks available to evaluate?\n\n2.How large finetuning dataset is required to train a sufficient model that internalized the CoT? Need more finetuning details.\n\n3.How does the finetuned model perform on unseen code generation tasks?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1.Comprehensive evaluation of CoT prompting strategies across different model sizes and benchmarks. I found the negative results that Reasoning prompts not additive interesting.\n\n2.Usage of pass n@k metric for fairer comparison between single-turn and multi-turn approaches.\n\n3.Thorough analysis of the impact of different error-feedback granularities.\n\n4.Investigation of fine-tuning on multi-turn CoT data and its effects on model behavior, showing effectiveness to internalize CoT steps for LLMs.", "weaknesses": "1.Although this work has solid empirical evaluations and good presentations, the novelty is a bit limited by combining existing works together, including CoT strategies, error-feedback granularities, and Pass n@k. The core contribution is primarily a prompt engineering effort combined with the evaluation of existing LLM capabilities in code generation. While valuable, this doesn't represent a significant theoretical or methodological advancement in the field. As for the RFT part, although I think it is a novel method used here, I would like to see more details on it to explain it clearly, with details on how it is finetuned with.\n\n2.Limited theoretical analysis of why certain prompting strategies work better than others. Although the authors presented solid empirical results on the prompting strategies ablation, I will still wonder why it works. Plus, since there is no best CoT for every case, i still think cases can be categorized into a bunch of properties, would it be better to have a task analysis model first, to decide which CoT to be applied based on these properties? I would be happy to see the answer to it.\n\n3.Potential overfitting concerns with fine-tuning on multi-turn CoT data generated from the same benchmark used for evaluation, since currently the training process is not clearly described, i keep this as a doubt and look forward to the authors' response.\n\n4.Figure presentations: I found some figures in the appendix not using vector figures, which are in low-resolution, starting from line 1100 and after. And some figures don't have proper captions, starting from line 1524 and after. The typesetting in the appendix should also be reorganized to avoid large blank pages. In a word, I think the good presentation in the main text should be kept as well in the appendix for a good ICLR paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)