What Makes Large Language Models Reason in (Multi-Turn) Code Generation?

## Abstract

Abstract Prompting techniques such as chain-of-thought have established themselves as a popular vehicle for improving the outputs of large language models (LLMs).
For code generation, however, their exact mechanics and efficacy are under-explored.
We thus investigate the effects of a wide range of prompting strategies with a focus on automatic re-prompting over multiple turns and computational requirements.
After systematically decomposing reasoning, instruction, and execution feedback prompts, we conduct an extensive grid search on the competitive programming benchmarks CodeContests and TACO for multiple LLM families and sizes (Llama 3.0 and 3.1, 8B, 70B, 405B, and GPT-4o).
Our study reveals strategies that consistently improve performance across all models with small and large sampling budgets.
We then show how finetuning with such an optimal configuration allows models to internalize the induced reasoning process and obtain improvements in performance and scalability for multi-turn code generation.

## 1 Introduction

The field of automatic code generation has made significant progress, particularly with the development of specialized Large Language Models (LLMs) (Chen et al., ; Li et al., ; Rozière et al., ; OpenAI, ; AI @ Meta, ).
While these models have demonstrated proficiency in generating simple functions across various programming languages, there is still considerable room for improvement in their ability to tackle more complex algorithmic reasoning tasks, such as those found in competitive programming benchmarks like CodeContests (Li et al., ).
Current state-of-the-art approaches either rely on model ensembling and massive single-turn sampling (AlphaCode Team, ) or employ complex structured prompt chains for planning, editing and debugging (Ridnik et al., ; Islam et al., ).
In contrast, multi-turn code generation strikes a balance between single-turn approaches and prompt chains, where code is built upon previous outputs in a dialog-like structure.
This approach is motivated by applications such as LLM-based agents (Yao et al., ), where models are tasked with decision-making and interacting with environments.
In code generation, multi-turn approaches have primarily been explored on simple benchmarks or in small sample regimes due to their association with self-repair techniques (Olausson et al., ; Chen et al., ; Shinn et al., ; Zhong et al., ).

In this paper, we systematically deconstruct the components of previous research on prompting techniques and propose a unified framework for multi-turn code generation.
Our objective is to establish a comprehensive and strong baseline, and to explore behavior and limitations across various sample regimes.
Our focus on competition-level coding benchmarks and sample budgets is motivated as follows:
(1) Popular methods such as chain of thought (Wei et al., , CoT) yield improvements on reasoning-heavy tasks. However, they are designed to elicit reasoning traces for maximizing single-turn performance and are not inherently multi-turn.
Competition-level benchmarks require algorithmic reasoning and thus provide an ideal testbed to evaluate whether CoT techniques can be extended beyond single-turn reasoning.
(2) Recent studies suggest that the performance gains from self-repair are often modest when considering their generation cost (Olausson et al., ) and that repeated single-turn sampling serves as a strong baseline (Brown et al., ).
As such, the trade-off between single-turn and multi-turn approaches, and the optimal allocation of resources between them, remains under-explored.

Figure: Figure 1: Our framework for evaluating LLM multi-turn code generation techniques. Top: In the default multi-turn setting, given a programming problem, the model generates a code solution, interacts with the runtime environment to gather execution feedback and retries in case of failure. Bottom: On top of the default setting, we gather *reasoning* (Reason.) prompts, *instruction* (Inst.) prompts, and *execution feedback* prompts. The problem statement is augmented with a *reasoning* prompt. After generating an answer to the *reasoning* prompt, an *instruction* prompt determines how program code should be generated. The *execution feedback* prompts vary in granularity, ranging from a binary pass or fail indicator to detailed tracing information.
Refer to caption: x1.png

Our framework (Figure 1) enables mix-and-match combinations of single- and multi-turn code generation and chain-of-thought (CoT) techniques(^1^11We use the term ”chain of thought” to refer to a broad family of prompting methods eliciting intermediate steps before or during code generation.): prompts that induce *reasoning*, such as a predicting problem attributes or writing natural language solutions first, and *instructions* that prompt different programming styles such as including comments or helper functions.
Finally, we integrate *execution feedback* from intermediate solutions to allow for code repair.
We conduct a comprehensive experimental survey across different benchmarks, LLM families and sizes, as well as sample regimes.
Our analysis yields several key insights:

- 1.
In the single-turn setting, combining *reasoning* prompts and *instruction* prompts achieves the best performance, and is more beneficial on larger models or harder problems. We also identify CoTs that degrade performance (Section 5.1).
- 2.
The multi-turn setting alone brings modest gains and is sometimes worse than its single-turn counterpart under equal sampling budgets. The combination with CoT provides a significant performance boost on all models we study.
Interestingly, detailed *execution feedback* prompts do not always translate to improved performance (Section 5.2).
We show that this can be attributed to reduced diversity of generated programs which results in performance drops for large sample budgets.
- 3.
LLMs can be instilled with reasoning behavior by finetuning on multi-turn CoT data (Section 5.3). The resulting model surpasses our best prompting configurations even without explicitly asking for CoTs during inference.

## 2 Background

### 2.1 Single-turn vs. Multi-turn Generation: Problem setting

We assume a coding problem $D=\{s,u,t\}$, where $s$ is the
problem
statement in natural language (e.g. see
Figure 1), $u$ is a
set of public tests, and $t$ is a set of private tests. A given code sample $c$
is considered correct if it passes *all* tests, or incorrect otherwise. Let $\pi$ denote an LLM that is able to
produce a code sample $c$ for $D$
from a user prompt $p$ which
includes the problem statement $s$.
In the single-turn setting we thus obtain a code sample $c\sim\pi(\cdot\ |\ p).$

In multi-turn code generation, we can distinguish between a *Natural-Language-to-Code* (NL $\rightarrow$ Code) task in the first turn and *Code-to-Code* (Code $\rightarrow$ Code) generation in subsequent turns.
For a given problem, we generate a sequence of intermediary code samples $c_{1},\ldots,c_{T}$ rather than just one.
After each turn $i$, the code sample $c_{i}$ is fed back into the model $\pi$ together with an *execution feedback* prompt to obtain the next sample $c_{i+1}$. This process is repeated $T$ times until we either pass all public tests or until a maximum number of turns $N$ is reached. More formally, we can obtain every intermediary sample $c_{i}$, including the final code solution $c_{T}$, as follows:

$$ $c_{i}\sim\pi(\cdot\ |\ p_{1},c_{1},p_{2},\dots,c_{i-1},p_{i}).$ $$

In this setting, the first prompt $p_{1}$ is the initial user prompt including the problem statement, and each $p_{i}$ for $i>1$ is an *execution feedback* prompt containing the runtime result with error information or traceback optionally attached.

In the remainder of this study, the sequence $(p_{1},c_{1},...,p_{T},c_{T})$ is denoted a *trajectory*, and the final code sample $c_{T}$ is called the *submission*. Only the code sample $c_{T}$ is tested against the private tests $t$ for correctness (i.e. intermediary code samples $c_{i}$ will only be tested against public tests $u$). Note that we sample not just one but several trajectories in parallel, starting with the same initial prompt $p_{1}$.

### 2.2 Evaluation Metrics

Figure: Figure 2: Scaling number of turns is not compute optimal. Pass@$10$ (Top) and pass $10$@$100$ (Bottom) on CodeContests test set when increasing the number of turns with Llama 3.1 70B.
Refer to caption: x2.png

We are interested in finding a correct solution to a given programming problem with a fixed budget, i.e., with a fixed number of code samples.
For estimating the success rate of generated code samples, pass@$k$ is a widely used metric (Chen et al., ). For a problem $P$ and given a budget of $k$ samples, pass@$k$ is the expectation that at least one sample is correct, i.e., that it passes all tests.

Limitations of pass@$k$
Pass@$k$ ignores computational requirements and thus puts single-turn evaluations at a disadvantage.
In multi-turn settings, solutions are obtained via several generations (i.e., LLM calls) and hence at a higher cost, rendering these two setups not directly comparable (Kapoor et al., ).

In this study, we opt to measure performance via pass $n$@$k$ (Li et al., ) rather than pass@$k$ for a fair comparison of techniques.
Pass $n$@$k$ estimates the success rate of a model $\pi$ on a problem $P$ using $k$ generations but at most $n$ submissions; it is the expectation that out of $n$ submissions one of them is correct (Appendix A).
Following Li et al. (), we select $n$ submissions based on public test performance.
Note that for $n=k$, both metrics are equivalent.
For each benchmark, we report the average pass $n$@$k$ or pass@$k$ over all problems.

Figure 2 compares pass@$k$ and pass $n$@$k$ when measuring performance in a multi-turn setting.
Pass@$10$ (Top) keeps increasing if we increase the maximum number of turns. However, pass $10$@$100$ (Bottom) shows that compute optimality is lost after 3 turns. Given a budget of 100 samples with 10 programs selected as submissions, the optimal allocation of compute is obtained by generating trajectories with 3 turns at most. As such, throughout this paper, we favor pass $n$@$k$ and report pass@$k$ only when comparing single-turn results exclusively.

## 3 Prompting and Feedback Space

Figure: Figure 3: Prompting space explored in our survey. We explore chain of thought prompts at three different levels: before the first code generation (*reasoning* prompts), with code generation (*instruction* prompts), and after the first code generation (*execution feedback*). The corresponding works from the single-turn and multi-turn reasoning and code generation literature are: [1] Gao et al. (), [2] Zhou et al. (), [3] Khot et al. (), [4] Zelikman et al. (), [5] Jain et al. (), [6] Zhong et al. (), [7] Ni et al. (), [8] Chen et al. (), [9] Le et al. (), [10] Madaan et al. (), [11] Paul et al. (), [12] Tang et al. (), [13] Li et al. ().
Refer to caption: x3.png

We map the space of prompting techniques studied in our experimental survey in Figure 3. As CoT can intervene at different times in code generation, we categorize *reasoning* prompts (NL $\rightarrow$ NL) that elicit understanding of the problem before code generation, and *instruction* prompts (NL $\rightarrow$ Code) that guide the code output to enhance readability and modularity. These prompts can be applied in single-turn and multi-turn approaches.

In the multi-turn setting, we also introduce *execution feedback* prompts directly harvested from the runtime environment, serving as additional information for the model to self-repair within turns. We aim to determine the type of feedback which most effective on competitive programming benchmarks in the large sample regime.
We thus evaluate several types of feedback, ranging in granularity:

- •
Binary feedback: A simple pass/fail indicator.
- •
Failed tests feedback: Provides expected and actual values for failed unit tests, along with tracebacks if any runtime errors are encountered.
- •
Failed & passed tests feedback: Expands on failed tests feedback by also including input/output information for passing tests.
- •
LDB feedback (Zhong et al., ): Offers debugger information, printing intermediate variable values and separating the code into blocks. The model must identify at which block the code failed and attempt to fix it.

CoT and execution feedback are incorporated into the generation through specific prompts as illustrated in Figure 1 (Bottom).
As we will show in Section 5.2, different types of *execution feedback* induce different multi-turn behavior that can be classified as either *exploratory* or *exploitative*.

## 4 Experimental Setting

Models    We perform experiments with the Llama Instruct series of LLMs, including Llama 3.0 and 3.1, 8B and 70B models (AI @ Meta, ). We use Llama 3.1 405B and GPT-4o in small sampling regimes only due to compute constraints.

Single-turn    Our grid search comprises 8 *reasoning* prompts and 6 *instruction* prompts, detailed in Appendix D. The *reasoning* prompts elicit intermediate steps either in natural language or with partial code. The *instruction* prompts either increase code readability ("describe"), break down the solution into modular code ("modularity"), or bias the type of solution ("solution"). Although we perform one more step of LLM inference for the *reasoning* prompts, we do not consider it an additional turn as our study compares the number of code attempts per problem and the effect of adding different types of extra tokens. We argue that this is equivalent to a single LLM call which groups all the *reasoning* prompts together, modulo the number of LLM forward passes. We generate with nucleus sampling (Holtzman et al., , top-p=0.95) and a temperature of $1.0$ to encourage output diversity.

Multi-turn
When performing multiple consecutive attempts at solving a coding problem, we set the code attempt limit to 3; this is motivated by the multi-turn results in Section 2.2, which reveal three turns as compute-optimal.
We take the best *reasoning* prompts from the single-turn setting and combine them for up to 3 *reasoning steps* before code generation.
We also introduce the *CoT-retry* setup, which allows for adaptive inference budget based on problem difficulty. In the first turn, we omit CoT prompts. If the first solution fails on more challenging problems, we prompt the LLM with a combination of *execution feedback* and a *reasoning* prompt. We employ a different prompt for each turn (see Appendix D.3). We also ablate different granularities of *execution feedback*.
We do not include CoT prompts in this feedback comparison to isolate the effect of different feedback types.

Rejection Sampling Finetuning
With the Llama 3.1 70B model, we use the *CoT-retry* strategy to generate 3-turn trajectories on the CodeContests training set. We filter out trajectories with incorrect final code and perform supervised finetuning on the resulting data (details in Appendix B.2).

Benchmarks    We conduct our experiments on two competitive coding benchmarks in the zero-shot setting: (1) CodeContests (Li et al., ) contains 13k programming problems in the training set and 117/165 problems in the valid/test set.
Each problem contains public tests, private tests, and generated tests.
We use public tests to provide execution feedback in the multi-turn setting and use all available tests to evaluate the final submission. (2) TACO (Li et al., )
is a collection of problems sourced from CodeContests, APPS (Hendrycks et al., ), and various programming contest platforms.
The test set is split into 5 distinct difficulty levels: easy, medium, medium-hard, hard, and very-hard, with each level comprising 200 problems. This stratification allows us to examine the performance of different prompting strategies across difficulty levels. We use the first test case as the public test.

## 5 Results

In this section, we first present model performance with the respective best CoT variants in Table 1 and 2 and then conduct a series of detailed experiments to better understand the performance impact of individual prompting methods.
We structure our presentation by key findings outlined in Introduction.

**Table 1: Up to +10% with multi-turn CoT. Pass $n$@$k$ (%) on CodeContests test set with high temperature (1.0) and large sampling budget. In the multi-turn setting, we use a maximum of 3 code attempts (i.e., 3 turns) with the "failed tests" feedback. The pass $n$@$k$ is calculated from 200 trajectories for both single-turn and multi-turn settings. We also report the pass rates for Llama 3.1 70B after Rejection Sampling Fine-tuning (RFT) (Section 5.3).**
| Model | Variants | CodeContests / Test |  |  |  |
| --- | --- | --- | --- | --- | --- |
| 1@3 | 10@30 | 33@100 | 100@300 |  |  |
| Llama 3.0 8B |  | 2.9 | 8.0 | 12.6 | - |
|  | + CoT | $3.4_{+0.5}$ | $11.7_{+3.7}$ | $\mathbf{17.3}_{\mathbf{+4.7}}$ | - |
|  | + Multi-turn | $2.4_{-0.5}$ | $8.0_{+0.0}$ | $12.8_{+0.2}$ | 16.7 |
|  | + Multi-turn CoT | $2.8_{-0.1}$ | $9.8_{+1.8}$ | $14.9_{+2.3}$ | 19.4 |
| Llama 3.0 70B |  | 9.6 | 18.9 | 23.1 | - |
|  | + CoT | $10.4_{+0.8}$ | $26.0_{+7.1}$ | $33.0_{+9.9}$ | - |
|  | + Multi-turn | $10.1_{+0.5}$ | $21.0_{+2.1}$ | $26.7_{+3.6}$ | 32.7 |
|  | + Multi-turn CoT | $11.1_{+1.5}$ | $26.5_{+7.6}$ | $\mathbf{{34.3}_{+11.2}}$ | 40.4 |
| Llama 3.1 8B |  | 7.7 | 18.2 | 23.8 | - |
|  | + CoT | $8.0_{+0.3}$ | $19.5_{+1.3}$ | $\mathbf{26.1_{+2.3}}$ | - |
|  | + Multi-turn | $7.0_{-0.7}$ | $18.8_{+0.6}$ | $24.5_{+0.7}$ | 30.4 |
|  | + Multi-turn CoT | $6.9_{-0.8}$ | $19.4_{+1.2}$ | $26.0_{+2.2}$ | 31.5 |
| Llama 3.1 70B |  | 24.1 | 42.3 | 49.8 | - |
|  | + CoT | $26.4_{+2.3}$ | $47.8_{+5.5}$ | $54.8_{+5.0}$ | - |
|  | + Multi-turn | $24.1_{+0.0}$ | $43.8_{+1.5}$ | $51.6_{+1.8}$ | 56.2 |
|  | + Multi-turn CoT | $27.7_{+3.6}$ | $48.4_{+6.1}$ | $\mathbf{55.3_{+5.5}}$ | 59.6 |
| $\text{Llama 3.1 70B}^{\text{RFT}}$ |  | 26.2 | 45.1 | 50.9 | - |
|  | + Multi-turn | $29.7_{+3.5}$ | $50.5_{+5.4}$ | $\mathbf{57.2_{+6.3}}$ | 61.1 |

**Table 2: Benchmarking of CoT across models: GPT-4o and Llama. Pass $1$@$1$ (%) and pass $1$@$3$ (%) with low temperature (0.2). As models become more capable, repeated sampling surpasses a straightforward extension to multi turn (e.g. GPT-4o) or single-turn CoT (e.g. Llama 3.1 405B). A tailored multi-turn CoT, however, improves pass $1$@$3$ performance across all models.**
| Variants | GPT-4o | Llama 3.1 70B | Llama 3.1 405B |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| 1@1 | 1@3 | 1@1 | 1@3 | 1@1 | 1@3 |  |
| Single-turn | 17.0 | 27.6 | 23.2 | 27.3 | 27.8 | 32.9 |
| + CoT | $25.5_{+8.5}$ | $29.0_{+1.4}$ | $25.5_{+2.3}$ | $28.9_{+1.6}$ | $25.1_{-2.7}$ | $31.8_{-1.1}$ |
| + Multi-turn | - | $23.1_{-4.5}$ | - | $29.5_{+2.2}$ | - | $35.4_{+2.5}$ |
| + Multi-turn CoT | - | $\mathbf{31.5_{+3.9}}$ | - | $\mathbf{31.5_{+4.2}}$ | - | $\mathbf{40.1_{+7.2}}$ |

### 5.1 Single-Turn Setting: CoT works best for hard problems, large models, high sampling

**Table 3: Combining *reasoning* and *instruction* works best as compared to each individually for single-turn CodeContests test set (chosen based on pass@$100$ performance per model). In the best categories, results worse than the baseline are underlined.**
|  | Llama 3.0 8B | Llama 3.0 70B | Llama 3.1 8B | Llama 3.1 70B |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | pass@1 | pass@100 | pass@1 | pass@100 | pass@1 | pass@100 | pass@1 | pass@100 |
| Baseline | 1.6 | 12.3 | 3.8 | 23.8 | 3.8 | 22.8 | 16.7 | 48.9 |
| Worst *reasoning* | 1.4 | 12.9 | 5.7 | 21.8 | 4.0 | 23.4 | 15.6 | 47.4 |
| Worst *instruction* | 1.4 | 11.3 | 3.4 | 25.1 | 3.7 | 20.9 | 14.9 | 48.4 |
| Worst Combination | 1.4 | 11.8 | 5.6 | 21.0 | 2.9 | 21.1 | 13.2 | 43.5 |
| Best *reasoning* | 1.8 | 15.7 | 7.0 | 30.4 | 4.1 | 25.7 | 15.7 | 52.2 |
| Best *instruction* | 1.3 | 13.5 | 5.5 | 29.6 | 3.6 | 24.6 | 16.8 | 53.8 |
| Best Combination | 1.5 | 17.3 | 5.3 | 33.1 | 4.0 | 26.1 | 16.1 | 54.1 |

We first investigate the impact of various CoT prompting strategies on models in the single-turn setting. There will be no *execution feedback* prompts. Therefore, our grid search involves searching in the space of *reasoning* prompts (NL $\rightarrow$ NL) and *instruction* prompts (NL $\rightarrow$ Code).

##### Reasoning and instruction prompts can work together.

We first compare the effect of various *reasoning* prompts, *instruction* prompts as well as combinations of both. Synthesized results are presented in Table 3, and we refer to Appendix C.1 for the complete set of experiments that led to Table 3. An interesting observation is that even the best performing *reasoning* and *instruction* prompts for pass@$100$ can decrease model performance in small sampling regimes (pass@$1$). Although *reasoning* prompts provide larger gains than *instruction* prompts (with the exception of Llama 3.1 70B), combining both results in the best performance.

Figure: Figure 4: CoT helps most on hard examples. From a set of 8 reasoning and 6 instruction prompts commonly used on competitive coding benchmarks, we extract the pass rate of the best and worst prompts amongst all $63=(8+1)\times(6+1)$ combinations (including no *reasoning* or no *instruction*) for Llama 3.0 8B. We compare on different difficulty split of the TACO dataset. The relative gain from a tailored CoT increases with problem difficulty and sampling size.
Refer to caption: x4.png

##### CoT is most helpful for large models.

With the smaller Llama 3.0 8B and Llama 3.1 8B, we observe from Table 3 that the best combination of *reasoning* and *instruction* prompts provides relatively small gains of 5.0% and 3.3% pass@$100$ on the CodeContests test set compared to the improvements of 9.3% and 5.2% from the corresponding 70B models. Interestingly, we found that not all sets of prompts are beneficial.
the worst combination degrades the pass@$100$ of Llama 3.1 70B by up to 5.4%. CoT makes performance worse if the model fails to follow the *instructions* or makes the LLM propose a sub-optimal plan. Sub-optimal plans are usually brute force approaches to solve the problem which do not fit the time limits constraint (see Appendix E for an example).

##### CoT is most helpful for harder problems.

With the TACO dataset, which provides a difficulty split,
we can observe that CoT does help smaller models on harder problems. Figure 4 demonstrates that the gap between baseline performance (No CoT) and the best *reasoning* and *instruction* prompt combination widens with problem difficulty. For example, the pass@$100$ of Llama 3.0 8B nearly doubles with CoT on the very-hard test split (2.1% $\rightarrow$ 3.9%). We show in Appendix C.3 that this observation generalizes to Llama 3.1 8B and 70B model.

##### Prompt efficacy is model and sample size dependent.

No singular reasoning and instruction combinations work best across sampling sizes and models (see Appendix C.2 for detailed analysis). Reasoning prompts that simplify the problem (e.g., self-reflection, explain input-output pairs) benefit smaller models (8B models) whereas larger models (70B, 405B, GPT-4o) gain most from generating parts of the solution (e.g., write function docstrings). "Solution"-based instruction prompts are the most efficient across models, specifically for the Llama 3.1 series, as shown in Figure 5.

Figure: Figure 5: Solution-based *instruction* prompts work best across Llama 3.1 models. We separate *instruction* prompts into "describe" (e.g., add comments, imports), "modularity" (e.g., add helper functions) and "solution"(e.g., write a naive solution, propose a clever algorithm). The performance difference ($\Delta$) is normalized with respect to the baseline and standard deviation per pass rate.
Refer to caption: x5.png

### 5.2 Multi-turn setting: self-repair limited without CoT and proper feedback

We summarize our multi-turn results in Table 1. With a fixed number of samples, i.e., $k$ in pass $n$@$k$, multi-turn alone provides modest gains only (usually less than +2%) and sometimes even reduces pass $1$@$3$ performance compared to drawing independent samples in single-turn mode.

Figure: Figure 6: Fine-grained feedback induces exploitative behavior. Distribution of consecutive code similarity scores within dialog for different types of feedback, obtained from Llama 3.1 8B and 70B samples (temperature 1.0). The higher the similarity scores between consecutive codes in the same dialog, the more the model exhibits exploitative behavior.
Refer to caption: x6.png

Notably, this is the case for smaller models (Llama 3.0 and 3.1 8B).
In this section, we take a closer look at performance
drops in the multi-turn setting and explore methods that can take advantage of accessing previous wrong solutions.

##### Reasoning prompts are not additive.

Figure: Figure 7: Reasoning and execution feedback prompts, and RFT, enhance both single- and multi-turn performance for Llama 3.1 70B.
Refer to caption: x7.png

It is tempting to consider that stacking more *reasoning* prompts before code generation will further guide the model towards correct solutions. For example, prompts might increase the granularity of reasoning: self-reflect on the problem, explain the input/output pairs, write helper functions, and finally output a full code solution. However, we empirically find that across models, one step of *reasoning* provides the most significant boost. The performance plateaus or even decreases with two or three steps. Increasing the number of *reasoning* steps hurts both Llama 3.0 and 3.1 models (see Table 5 in Appendix C.4). For the best models, a single step with a *reasoning* prompt is most beneficial.

##### CoT-retry works best.

For Llama 3.0 models, simply extending the single turn *reasoning* and *instruction* prompts to the multi-turn setting yields superior performance (reported as "Multi-turn CoT" in Table 1). However, as models become more capable, an increasing number of problems in CodeContests are solved in the first attempt without specific prompts. *CoT-retry* only *reasons* when the first attempt fails and therefore works best across Llama 3.1 models for all sampling sizes and benchmarks ("Multi-turn CoT" in Table 1). Figure 7 decomposes its per-turn performance. When extending the number of turns from 2 to 3, Llama 3.1 70B alone shows diminishing gain while combination with *CoT-retry* still increases the performance by a large margin.

##### Execution feedback granularity determines exploration-exploitation behavior.

Given previous incorrect code and execution feedback, subsequent attempts can consist of a fresh attempt (exploration) or of updates to prior solutions based on feedback (exploitation).
We quantify this behavior by computing similarity scores between two consecutive solutions (details in Appendix B.1). Figure 6 shows that with more fine-grained information provided via execution feedback, models exhibit exploitative behavior (high similarity scores). Exploitation can be a desired property on relatively easy problems where errors are due to simple bugs. However, we posit that diversity is key to improving performance on difficult problems, i.e., exploratory behavior within a trajectory based on the *execution feedback* prompts. This matches our experimental results: simple execution feedback (e.g., binary, failed tests) provides optimal performance for most models (Appendix C.6).

### 5.3 CoT Rejection Sampling Fine-tuning: models can internalize reasoning

**Table 4: Multi-turn CoT and RFT generalize to TACO test set. Pass $n$@$k$ (%) of Llama 3.1 70B on multi-turn TACO test set with temperature 1.0. We use the best multi-turn CoT found on CodeContests. We use the model RFTed on CodeContests training set (after decontamination, details in Appendix F) and report its performance directly on TACO without CoT.**
| Model | easy | medium | medium_hard | hard | very_hard |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1@3 | 100@300 | 1@3 | 100@300 | 1@3 | 100@300 | 1@3 | 100@300 | 1@3 | 100@300 |  |
| Llama 3.1 70B | 31.6 | 60.2 | 14.2 | 44.6 | 9.5 | 36.2 | 4.4 | 20.6 | 1.8 | 9.0 |
| + Multi-turn CoT | 32.3 | 59.8 | 15.0 | 46.2 | 10.8 | 38.5 | 5.8 | 22.8 | 2.6 | 11.8 |
| $\text{Llama 3.1 70B}^{\text{RFT}}$ | 34.1 | 58.9 | 18.0 | 45.3 | 13.0 | 39.4 | 8.1 | 23.3 | 3.5 | 12.0 |

Figure: Figure 8: RFT makes the model produce more diverse code within trajectories as shown by the consecutive codes’ similarity scores before/after RFT on CodeContests test set evaluated with multi-turn no CoT. This shift towards more exploratory behavior contributes majorly to the gain of correct trajectories.
Refer to caption: x8.png

We investigate whether LLMs can benefit from finetuning on reasoning traces obtained via CoT prompting.
We thus perform Rejection Sampling Finetuning (RFT)
on Llama 3.1 70B, where the *reasoning*, *instruction* and
*execution feedback*
prompting strategies we consider act as *policy improvement operators*:
they elicit the model’s reasoning ability and produce a higher number of trajectories

Figure: Figure 9: Llama 3.1 70B’s pass $k$@$3k$ on CodeContests. *CoT-retry* increases the performance in large sampling regimes. RFT transfers this reasoning ability to no CoT setting and lifts the pass rate curve across sampling budgets.
Refer to caption: x9.png

with correct submissions.
Given the low variance across different feedback types (Table 7
in Appendix C.6),
we opt for simplicity and use the "failed tests" execution feedback combined with *CoT-retry* for data generation.

More specifically, we improve a model $\pi$ by 1) collecting a dataset of correct trajectories sampled from $\pi$ with CoT enabled at inference time, 2) removing the CoT prompt in the collected trajectories, and 3) finetuning $\pi$ with the standard next-token prediction objective. With this strategy, we can now obtain CoT-level trajectories without adding specific prompts at inference time.

Figure 9, Table 1, and Table 4 show that the RFT model provides additional gains over inference methods across sampling sizes and datasets. Beyond performance, RFT on multi-turn CoT improves sampling diversity (Figure 8) and self-repair capacities, especially for long trajectories (Figure 7).
Behavior-wise, we show in Table 6 (Appendix C.5) that RFT results in model responses with increased textual content.

## 6 Related Work

Chain of Thought with Code    Chain of Thought (CoT)
enables step-by-step thinking for LLMs to solve mathematical word problems in either few-shot (Wei et al., ) or zero-shot (Kojima et al., ) settings. Many variants, e.g., Tree of Thought (Yao et al., ), have emerged in code generation since. Chen et al. () and  Gao et al. () translate natural language mathematical problems in executable code for the model to separate reasoning and computation. These methods rely on the LLM outputting correct code to represent a problem. We see this work as tangential to ours as boosting LLM coding performance will also help on overall reasoning tasks. Higher levels of abstractions (Khot et al., ; Zhou et al., ; ; Zelikman et al., ; Jain et al., ) and self-repair techniques (Paul et al., ; Li et al., ; Ridnik et al., ) have been proposed. Beyond inference methods,Wadhwa et al. (); Yu et al. (); Zelikman et al. (); Hosseini et al. (); Pang et al. () explore new training algorithms and loss functions to learn from CoT. In comparison, we bring novelty to the type of CoT used in training (multi-turn) and rely on simple Rejection Sampling Fine-tuning (RFT) (Touvron et al., ; Yuan et al., ; AI @ Meta, ). It has been shown to achieve good performance, with less data compared to SFT (Setlur et al., ).

Execution feedback    Currently LLMs struggle to understand code execution feedback (Gu et al., ) as this type of data is rarely present in their training set. Zhong et al. () and  Ni et al. () try to mimic "print debugging" to convey intermediate code steps to the LLM.
Olausson et al. () found that the effect of self-repair largely depends on the text quality of the subsequent reasoning and therefore use only textual feedback. In our setting, we are interested in the feedback which could be directly harvested from the execution environment.
Shi et al. (); Li et al. (); Chen et al. () likewise proposed unit test generation as a way to increase coverage with execution feedback. Adding test generation to our pipeline would be an interesting avenue for further work.

Inference Optimization
With the rise of LLM agents (Kapoor et al., ) and the scaling effect of test time techniques (Li et al., ; Snell et al., ; Brown et al., ), inference optimization against compute resources becomes increasingly relevant.
Similar to our pass $n$@$k$ argument in Section 2.2,  Kapoor et al. () discuss the importance of controlling for generation cost in AI agent evaluations.

## 7 Limitations

In our multi-turn setting, we do not explore further branching at the second or third turn, i.e., more complex tree structures (Tang et al., ) or in general inference-based search approaches (Snell et al., ), e.g. with look-ahead or backtracking, as we focus on the effect of additional CoT tokens generation.
Although a maximally fair comparison (at the cost of complexity) should account for total input and output tokens (Olausson et al., ) as well as model size (Hassid et al., ), we believe pass $n$@$k$, which stresses the number of code attempts, constitutes a simple yet superior alternative to pass@$k$.
Our RFT is similar to Expert Iteration (Anthony et al., ) and ReST (Gulcehre et al., ) when considering a single iteration only. Future work could benefit from exploring more advanced training strategies in conjunction with multi-turn CoT to potentially further enhance reasoning capabilities. Finally, we speculate that the effectiveness of different prompts for different LLM families (particularly the Llama 3.0 vs. 3.1 series vs. GPT-4o) could be attributed to the mixture of finetuning data  (Chung et al., ). Exploration of this topic is beyond the scope of this paper.

## 8 Conclusion

In this work, we present a comprehensive experimental survey on various *reasoning*, *instruction* and *execution feedback* prompts in the single-turn and multi-turn code generation task at scale. Our results on two competitive programming benchmarks, CodeContests and TACO, suggest that incorporating CoT techniques, originally designed for single turns, and *execution feedback* prompts into the multi-turn setting is non-trivial. Due to the difficulty of the benchmarks, a major contributor to performance is problem understanding rather than the ability to perform code repair with detailed feedback. With a set compute budget, using multiple turns alone can hamper performance compared to repeated sampling with high temperatures. Biasing the model with adapted CoT based on problem difficulty at each turn boosts its self-repair abilities and leads to consistent gains across all model series and sizes. Beyond inference methods, our RFT experiment shows that multi-turn reasoning traces triggered by prompts can be internalized, which leads to advanced reasoning abilities. We hope that our findings motivate further research in more advanced multi-turn settings. One example is repository-level code agents, where models interact with complex environments to gather feedback and extensive planning and reasoning capabilities are demanded.

#### Reproducibility Statement

As our paper focuses on inference methods with existing models, the key components for reproducibility are access to models, datasets, and prompt descriptions. All the models (except our fine-tuned RFT model) used in this paper are publicly available at the time of writing: Meta Llama 3.0 and 3.1 series are open-weight, and gpt-4o-2024-05-13 (GPT-4o in the paper) are available through OpenAI API. The two benchmarks we use: CodeContests ([https://github.com/google-deepmind/code_contests](https://github.com/google-deepmind/code_contests)) and TACO ([https://github.com/FlagOpen/TACO](https://github.com/FlagOpen/TACO)) are publicly available. We provide a complete list of all our prompts in Appendix D to reproduce single-turn and multi-turn experiments. We present the details of computing similarity score with normalization in Appendix B.1. Regarding finetuning, our main contribution relies on the data augmentation technique on CodeContests for which we present the details in data collection, deduplication, and decontamination approach, as well as statistics such as the number of trajectories and the number of total tokens in Appendix B.2,  C.8 and F. We detail our finetuning hyperparameters in Appendix B.2 to reproduce our RFT model training. We will release the code for our multi-turn and CoT methods to facilitate reproduction.

#### Acknowledgement

We thank Quentin Carbonneaux, Baptiste Rozière, Jade Copet, Olivier Duchenne, Fabian
Glöeckle, Badr Youbi Idrissi, Nicolas Usunier, Sten Sootla, Chris Cummins, Sida Wang, Pierre Chambon, Matthieu Dinot, Ori Yoran, Kush Jain, Naman Jain and all the members in FAIR CodeGen team for helpful technical contributions, suggestions, and insightful discussions. We thank the Infra team for the support for enabling a seamless compute cluster experience.

## Appendix A Formula and Algorithm for pass n 𝑛 n italic_n @ k 𝑘 k italic_k metrics

Formally, let $N$ be the total number of code samples. Let $F$ be the number of codes filtered by public tests, among which there could be false positives. Let $C$ be the number of correct codes that pass all the unit tests. The pass $n$@$k$ for a benchmark of problems is defined as follows:

$$ $\text{pass}\hskip 2.84526ptn\text{@}k=\mathbb{E}_{\text{Problems}}\left[1-\sum _{i=0}^{k}\left(\frac{\binom{F}{i}\binom{N-F}{k-i}}{\binom{N}{k}}\right)\left( \frac{\binom{F-C}{n_{p}}}{\binom{F}{n_{p}}}\right)\right],$ (1) $$

where $n_{p}=\min(i,\ n)$.

##### Explanation

The first term $\frac{\binom{F}{i}\binom{N-F}{k-i}}{\binom{N}{k}}$ is the probability of having $i$ filtered solutions among $k$ solutions, which obeys a hyper-geometric distribution, $\textsc{Hypergeometric}(F,\ N-F,\ k)$. Given the number of submissions $n_{p}=\min(i,\ n)$, the second term $\frac{\binom{F-C}{n_{p}}}{\binom{F}{n_{p}}}$ is the probability of having none of the correct solutions.

In evaluation, instead of computing the combinatorial number, we use monte-carlo estimation by re-sampling $k$ solutions $n_{\text{boot}}$ times for bootstrapping (in our case we use 10000). The algorithm for such is described in detail in Appendix A.3 of the Alphacode paper (Li et al., ).

## Appendix B Experiment Details

### B.1 Computing Similarity Score

We compute the similarity score of two python code snippets as follows.

First, we pre-process the code snippet to remove formatting and variable naming effects. We normalize variable names by running an in-order indexing scheme on the Abstract-Syntax-Tree (AST), as shown in Figure 10, followed by a simple formatting by lambda x: ast.unparse(ast.parse(x)). We note that there are 1%-2% of codes failing the parsing because of syntax error, in which case we skip this normalization step.

Figure: Figure 10: Example of variable renaming AST pass.
Refer to caption: x10.png

Second, we use difflib.SequenceMatcher to compute the similarity score for the normalized snippets.

### B.2 Fine-tuning Setting

Using *CoT-retry*, we generate $200$ code trajectories with a maximum of 3 code attempts. Note that if the model solves the problem in the first turn, the trajectory will still be collected while there will not be *execution feedback*.
After filtering the incorrect trajectories, we keep only $60\%$ the ones where the code in the last turn passes all the tests. We assume that correct final code correlates with correct reasoning in the CoT and self-repair techniques. We apply additional post-processing to the traces by removing the CoT prompt introduced, enabling the model to develop inherent CoT-like reasoning capabilities through fine-tuning. We separate the successful trajectories into 2 sets, single-turn trajectories and multi-turn trajectories, and conduct LSH-based deduplication on each set to the code solutions per problem instance to a maximum of 50 solutions.
We perform self-supervised fine-tuning on the multi-turn trajectories, using standard cross-entropy loss on the last full body of model response. The finetuning uses learning rate 2e-6, 545 steps of gradient updates, sequence length 8192, global batch size 524288 tokens, cosine scheduling with 10 warmup steps, and annealing to 10% of peak learning rate.

## Appendix C Additional Experiment Results

### C.1 Single-Turn Grid Search Results

We provide the complete grid search results for all our *reasoning* and *instruction* prompts across all models and pass rates for the single turn setting. This demonstrates the variability in effectiveness per sampling size and LLM series.
The "weak solution" *instruction* prompt is a clear winner for larger sampling sizes $k\geq 10$. We show in Figure 11, 12, 13 and 14 the grid search of all *reasoning* and *instruction* prompts for the Llama 3.0 and 3.1 series. As we increase the pass@, we increase the sample diversity and the recall across all CoT. For low pass@, most prompts hurt performance. CoT is the most effective with Llama 3.0 70B.

Figure: Figure 11: Grid search of all *reasoning* and *instruction* prompts for Llama 3.1 8B.
Refer to caption: x11.png

Figure: Figure 12: Grid search of all *reasoning* and *instruction* prompts for Llama 3.0 8B.
Refer to caption: x12.png

Figure: Figure 13: Grid search of all *reasoning* and *instruction* prompts for Llama 3.1 70B.
Refer to caption: x13.png

Figure: Figure 14: Grid search of all *reasoning* and *instruction* prompts for Llama 3.0 70B.
Refer to caption: x14.png

Figure: Figure 15: No gold CoT across models. Based on our grid search of instruction and reasoning prompts, we compare all 63 single-turn results across three different models. With a low sampling budget, most prompts perform similarly, if not worse than the baseline performance (without CoT). The best prompt (in green) differs for each model, but we see similar patterns in the Llama models.
Refer to caption: x15.png

### C.2 Detailed Analysis of Single-Turn Prompts

When comparing *reasoning* and *instruction* prompts, the values are normalized with respect to the baseline in each respective pass rate specifically: $x\leftarrow\frac{x-\text{baseline}}{\text{std}(\mathbf{x})}$. The value at 0 0, therefore, corresponds to no *reasoning* and no *instruction* prompts. We provide further results aggregated across models and types of prompts. As demonstrated by 17 and 18, we have large variations across models and prompt types and observe that no *reasoning* and *instruction* prompt always performs above the 0 baseline. As shown in Figure 16, the best combinations often rely on "weak solution" *instruction* but vary across sample sizes for *reasoning* with "self-reflection" for lower pass@s and "helper functions" for higher pass@s. We observed writing intermediate variables before code often made performance worse and could be qualified as the "worst" *reasoning* prompt for all models.

Figure: Figure 16: For pass@1, we average over all $6$ models (3.0 8B, 70B, 3.1 8B, 70B, 405B and GPT-4o) and pass@10 and pass@100 over 4 models (Llama 3.0, 3.1 8B and 70B). We take the mean across all results and plot the top 3 means per pass@.
Refer to caption: x16.png

Figure: Figure 17: Group by *instruction* prompts averaged across all *reasoning* prompts for the Llama 3.0 and 3.1 models. We observe that "check constraints" is a winner for pass@1 and "weak solution" for pass@100. Overall, "add a comment before each line" seems the least efficient across models.
Refer to caption: x17.png

Figure: Figure 18: Group by *reasoning* prompts averaged across all *instruction* prompts (top) for small models and (bottom) for large models. For pass@1, "explain IO pairs" helps small models, and "helper function docstrings" helps large ones. The relative efficacy of each prompt converges to a similar order for pass@100 for large and small models.
Refer to caption: x18.png

### C.3 Generalization of Single-Turn Best CoT to Llama3.1 models

We show in Figure 19 and 20 that the best CoT (i.e. *reasoning* and *instruction* prompt and their combination) found with Llama 3.0 8B on TACO could be directly ported to Llama 3.1 8B and 70B models. We also observe the effect that CoT brings more boost on harder problems by comparing the gain on the easy and very-hard split.

Figure: Figure 19: We use the best CoT (i.e., *reasoning* and *instruction* prompt combination) found with Llama 3.0 8B and test it directly with Llama 3.1 8B on the easiest (easy) and the most difficult (very-hard) split of TACO.
Refer to caption: x20.png

Figure: Figure 20: We use the best CoT (i.e., *reasoning* and *instruction* prompt combination) found with Llama 3.0 8B and test it directly with Llama 3.1 70B on the easiest (easy) and the most difficult (very-hard) split of TACO.
Refer to caption: x21.png

### C.4 Reasoning Prompts not Additive

We describe methods that did not help enhance multi-turn CoT, specifically adding more complex execution feedback and more steps of reasoning prompts. Our experiment result is shown in Table 7 that before outputting the first code, stacking more reasoning steps hurt the performance, especially for Llama 3.1 70B.

**Table 5: Stacking more *reasoning* prompts can hurt performance for Llama 3.1 70B.**
| Number of prompts | Llama 3.0 70B | Llama 3.1 70B |  |  |
| --- | --- | --- | --- | --- |
| 1@3 | 100@300 | 1@3 | 100@300 |  |
| 1 *reasoning* $\times$ 1 *instruction* | 11.2 | 40.0 | 24.5 | 59.2 |
| + 1 *reasoning* | -0.4 | -1.8 | -2.0 | -3.1 |
| + 1 *instruction* | -0.1 | +0.4 | -4.0 | -2.1 |

### C.5 RFT Model Behavior Analysis

We show in Table 6 the fraction of text characters by the total response length. We take into account the intermediary CoT response if CoT is used. RFT model significantly increases the text output around the code output, which could contain reasoning traces.

**Table 6: Fraction of text characters (not extracted as code) by the total response length. We also count the CoT response when CoT is enabled. RFTed model outputs more text in the response.**
| Model | Non-Code<br>Fraction |
| --- | --- |
| Llama 3.1 70B | 0.37 |
| + Multi-turn CoT | 0.57 |
| $\text{Llama 3.1 70B}^{\text{RFT}}$ | 0.50 |

### C.6 Simple Execution Feedback is Sufficient

We show in Table 7 that *execution feedback* prompts with different granularity present low variance with respect to the pass rate, both in high-temperature setting (1.0, pass $100$@$300$) and low-temperature setting (0.2, pass $1$@$3$).

We posit that for hard problems presented in the competitive programming benchmark, models generate wrong code not because the code is buggy by accident but because models do not understand how to solve the problem correctly.
It highlights the fact that for competitive programming benchmark, algorithmic reasoning (to align what the models believe to be a correct solution with the ground-true solution), as elicited by CoTs, impacts the performance more than bug-fixing ability (to align the emitted code with what the models believe to be a correct solution).

**Table 7: Execution feedback result on multi-turn CodeContests test set. Results are reported using 3-turn trajectories. $1$@$3$ is estimated from 20 trajectories per problem under temperature 0.2. $100$@$300$ is estimated from 200 trajectories per problem under temperature 1.0.**
| Feedback | Granularity | Llama 3.1 70B | Llama 3.1 8B |  |  |
| --- | --- | --- | --- | --- | --- |
| 1@3 | 100@300 | 1@3 | 100@300 |  |  |
| Binary | + | 28.8 | 55.9 | 10.9 | 30.9 |
| Failed tests (default) | ++ | 29.5 | 56.2 | 10.9 | 29.5 |
| Failed & passed tests | ++ | 29.5 | 55.0 | 10.7 | 30.4 |
| LDB (Zhong et al., ) | +++ | 26.5 | 54.8 | 9.9 | 29.1 |

### C.7 Ablation of Normalization Step in Similarity Score

We show in Figure 21 the distribution and histogram of similarity score without the normalization step. The similarity score, therefore, measures the raw code generated by the LLM. Compared with Figure 6 and 8, the fundamental trend does not change. The robustness against our normalization step shows that the LLMs we study output coherent (in terms of variable naming and formatting) code within the same dialog.

Figure: Figure 21: Distribution of consecutive code similarity scores (without the normalization step) when varying the *execution feedback* granularity.
Refer to caption: x22.png

Figure: Figure 22: Histogram of the similarity scores (without the normalization step) of consecutive codes generated by the model before/after multi-turn CoT RFT on CodeContests test set.
Refer to caption: x23.png

### C.8 Ablation of RFT Data Mixture

As detailed in Appendix B.2, we collect 2 sets of correct trajectories, single-turn (ST) and multi-turn (MT), on CodeContests training set and perform LSH-based deduplication to a maximum of 50 solutions (in each set) per problem statement. We also decontaminate the 2 sets from TACO test set as detailed in Appendix F.

After deduplication and decontamination, we harvest 177475 trajectories (in total 143M tokens) and 160600 multi-turn trajectories (in total 285M tokens). We show the ablation of the data mixture:

- •
train on multi-turn (MT) trajectories only
- •
train on single-turn (ST) and multi-turn (MT) trajectories

We show in Table 8 the ablation result. We find that training on multi-turn data only provides better performance. We hypothesize that the single-turn trajectories solve the problems of which models are already capable. Further reinforcement on these problems could potentially lead to overfitting and bias the model behavior towards trying to solve the problems in the first turn instead of enhancing its multi-turn capability.

**Table 8: Ablation of RFT data mixture. We show the best performance of two data mixture configurations: training on multi-turn (MT) data only and training on single-turn (ST) and multi-turn (MT) data.**
| RFT Data | CodeContests / Test |  |  |
| --- | --- | --- | --- |
| 1@3 | 10@30 | 100@300 |  |
| MT | 29.7 | 50.5 | 61.1 |
| ST + MT | 29.1 | 49.6 | 60.0 |

## Appendix D Prompts

We list the prompts used throughout our experiments inspired by recent works in code generation (Zelikman et al., ; Jain et al., ; Paul et al., ; Ridnik et al., ). We focus on zero-shot prompting techniques specific to competitive programming problems or, more general, to code generation. We classify prompts into two categories: reasoning and instruction. To determine this list, we ran experiments at a small scale (pass@$10$) with over $30$ prompts on $500$ examples sampled from the CodeContest training set. We picked the most promising ones in terms of final unit test pass and execution rates. Some of our prompts are adapted from recent works in competitive programming.

### D.1 Reasoning prompts

- •
Adapted from AlphaCodium Ridnik et al. ()
–
self-reflection: Given the code contest problem, reflect on the problem, and describe it in your own words, in bullet points. Pay attention to small details, nuances, notes and examples in the problem description.
–
predict IO pairs: Given the code contest problem and the provided examples, take the first 3 examples and explain how its input leads to the corresponding output. Read carefully the problem description. Make sure the test explanations are consistent with them, and between themselves. The explanation must coherently and logically lead from the input to the output. Be succinct.
–
write code solution with guidelines: Your goal is to come up with possible solutions to the code contest problem. Guidelines: Make sure each solution fully addresses the problem goals, constraints, examples, and notes. Each solution must have reasonable runtime and memory complexity - less than three seconds on a modern computer, given the problem constraints for large inputs. Double-check the solutions. Each possible solution must be able to generalize to additional test cases, not just the ones provided in the problem description.
- •
predict problem tag: Explain which two tags from the following list best apply to this problem: combinatorics, dynamic programming, math, bitmasks, number theory, brute force, data structures, divide and conquer, graphs, greedy, depth first search and similar, implementation, binary search, two pointers, strings, constructive algorithms, sortings, trees, disjoint set union.
- •
predict problem difficuly: Given the code contest problem, your task is to evaluate the difficulty of the problem either easy, medium or hard. Explain the difficulties of the problem and potential edge cases.
- •
write natural language solution: Generate a naive solution to this problem in natural language and then explain how you could improve it.
- •
write helper function docstring: Explain which helper functions you will need to solve the code contest problem. Without implementing them, write their signature and a doc string explaining their purpose.
- •
write intermediate variables and type: Explain what necessary intermediate variables you will need to solve the problem, specify their type and purpose in your solution.

### D.2 Instruction prompts

- •
Adapted from AlphaCodium Ridnik et al. ()
–
use helper functions: Guidelines: You must divide the generated code into small sub-functions, with meaningful names and functionality. Variables names should also be meaningful.
–
double check the import, variable, constraints: Double-check the solution code. Make sure to include all the necessary module imports, properly initialize the variables, and address the problem constraints.
- •
comment before each line: Write a comment before each line of code to explain your thought process.
- •
docstring before each function: Write a doc string before each function generated explaining its utility and expected inputs and outputs.
- •
generate weak solution and a second better one: Generate one solution in python, identify its weaknesses and then generate a second better algorithm to solve the problem.
- •
step by step: Think step by step and propose a clever algorithm.

### D.3 CoT-retry prompts

For a given problem, the model first solves it without any reasoning. If the first code generation is incorrect, we add an instruction prompt "write code solution with guidelines," to separate problem-solving and code writing. If the second code generation is false, we add another instruction prompt "generate a naive solution". These prompts are combined with the default (failed tests) execution feedback after each turn. This setup provides up to +6% improvements for the Llama 3.1 series. We show an example extracted from CodeContests generated by Llama 3.1 70B on the following page.

## Appendix E Examples Wrong CoT

Some problems are solvable by the model without CoT in the single-turn setting but become wrong when we add specific *reasoning* and *instruction* prompts. Some *instruction* prompts can lead to incorrect planning. The weak solution prompt, although effective when done correctly as shown by its high performance across models, can bias the model towards a brute force approach that it is unable to recover from as shown in example 1 and 2. These examples are generated with Llama 3.1 8B and extracted from CodeContests.

## Appendix F De-contamination between CodeContests and TACO

We found that there is a non-zero overlapping between CodeContests training set and TACO test set. Therefore, after gathering successful trajectories from Llama 3.1 70B on CodeContests training set, we further conduct de-contamination to filter out solutions to the problems that overlap with problems in TACO test set. We mined the contaminated problems as follows.

We note that exact string matching will result in a lot of contamination remaining undetected due to the different latex parsing and format between benchmarks.
We, therefore, use an off-the-shelf sentence embedding model to compute sentence similarity between problem statements from CodeContests training set and TACO test set. For each problem $P_{\text{taco}}$ in TACO test set, we set the threshold of sentence similarity to 0.8 to obtain similar CodeContests problems $\left\{P_{\text{CodeContests}}\right\}$. We take the first 5 solutions from $P_{\text{taco}}$ and run each solution against all the unit tests available of each similar problem $P_{\text{CodeContests}}$. If any of the solutions passes the unit tests, we label this as a contamination.

Our dataset mined from the Llama 3.1 70B output on CodeContests comprises solutions to 7238 problems in the training set. We detect 288 problems contaminated with the TACO test set, resulting in solutions to 6950 problems after filtering. This process further removes, after the LSH-based de-duplication, a total of 6422 entries from the single-turn trajectories and 7463 entries from the multi-turn trajectories.

We show an example of a contaminated problem in CodeContests training set and TACO test set below.

## Appendix G Contamination of TACO training set and test set

We also find that there are non-zero overlaps between TACO training set and test set. These overlaps, despite having different URL, have near identical problem statement. We find that this could be attributed to the fact that on the Codeforces platform, harder problems from easy contest (div2) could appear also in harder contest (div1) as easier problems. We show an example below, in which in training set the problem URL is [https://codeforces.com/problemset/problem/841/C](https://codeforces.com/problemset/problem/841/C) and in test set it is [https://codeforces.com/problemset/problem/840/A](https://codeforces.com/problemset/problem/840/A).

## Appendix E Examples Wrong CoT

Some problems are solvable by the model without CoT in the single-turn setting but become wrong when we add specific *reasoning* and *instruction* prompts. Some *instruction* prompts can lead to incorrect planning. The weak solution prompt, although effective when done correctly as shown by its high performance across models, can bias the model towards a brute force approach that it is unable to recover from as shown in example 1 and 2. These examples are generated with Llama 3.1 8B and extracted from CodeContests.

## Appendix F De-contamination between CodeContests and TACO

We found that there is a non-zero overlapping between CodeContests training set and TACO test set. Therefore, after gathering successful trajectories from Llama 3.1 70B on CodeContests training set, we further conduct de-contamination to filter out solutions to the problems that overlap with problems in TACO test set. We mined the contaminated problems as follows.

We note that exact string matching will result in a lot of contamination remaining undetected due to the different latex parsing and format between benchmarks.
We, therefore, use an off-the-shelf sentence embedding model to compute sentence similarity between problem statements from CodeContests training set and TACO test set. For each problem $P_{\text{taco}}$ in TACO test set, we set the threshold of sentence similarity to 0.8 to obtain similar CodeContests problems $\left\{P_{\text{CodeContests}}\right\}$. We take the first 5 solutions from $P_{\text{taco}}$ and run each solution against all the unit tests available of each similar problem $P_{\text{CodeContests}}$. If any of the solutions passes the unit tests, we label this as a contamination.

Our dataset mined from the Llama 3.1 70B output on CodeContests comprises solutions to 7238 problems in the training set. We detect 288 problems contaminated with the TACO test set, resulting in solutions to 6950 problems after filtering. This process further removes, after the LSH-based de-duplication, a total of 6422 entries from the single-turn trajectories and 7463 entries from the multi-turn trajectories.

We show an example of a contaminated problem in CodeContests training set and TACO test set below.

## Appendix G Contamination of TACO training set and test set

We also find that there are non-zero overlaps between TACO training set and test set. These overlaps, despite having different URL, have near identical problem statement. We find that this could be attributed to the fact that on the Codeforces platform, harder problems from easy contest (div2) could appear also in harder contest (div1) as easier problems. We show an example below, in which in training set the problem URL is [https://codeforces.com/problemset/problem/841/C](https://codeforces.com/problemset/problem/841/C) and in test set it is [https://codeforces.com/problemset/problem/840/A](https://codeforces.com/problemset/problem/840/A).

## Appendix F De-contamination between CodeContests and TACO

We found that there is a non-zero overlapping between CodeContests training set and TACO test set. Therefore, after gathering successful trajectories from Llama 3.1 70B on CodeContests training set, we further conduct de-contamination to filter out solutions to the problems that overlap with problems in TACO test set. We mined the contaminated problems as follows.

We note that exact string matching will result in a lot of contamination remaining undetected due to the different latex parsing and format between benchmarks.
We, therefore, use an off-the-shelf sentence embedding model to compute sentence similarity between problem statements from CodeContests training set and TACO test set. For each problem $P_{\text{taco}}$ in TACO test set, we set the threshold of sentence similarity to 0.8 to obtain similar CodeContests problems $\left\{P_{\text{CodeContests}}\right\}$. We take the first 5 solutions from $P_{\text{taco}}$ and run each solution against all the unit tests available of each similar problem $P_{\text{CodeContests}}$. If any of the solutions passes the unit tests, we label this as a contamination.

Our dataset mined from the Llama 3.1 70B output on CodeContests comprises solutions to 7238 problems in the training set. We detect 288 problems contaminated with the TACO test set, resulting in solutions to 6950 problems after filtering. This process further removes, after the LSH-based de-duplication, a total of 6422 entries from the single-turn trajectories and 7463 entries from the multi-turn trajectories.

We show an example of a contaminated problem in CodeContests training set and TACO test set below.

## Appendix G Contamination of TACO training set and test set

We also find that there are non-zero overlaps between TACO training set and test set. These overlaps, despite having different URL, have near identical problem statement. We find that this could be attributed to the fact that on the Codeforces platform, harder problems from easy contest (div2) could appear also in harder contest (div1) as easier problems. We show an example below, in which in training set the problem URL is [https://codeforces.com/problemset/problem/841/C](https://codeforces.com/problemset/problem/841/C) and in test set it is [https://codeforces.com/problemset/problem/840/A](https://codeforces.com/problemset/problem/840/A).

## Appendix G Contamination of TACO training set and test set

We also find that there are non-zero overlaps between TACO training set and test set. These overlaps, despite having different URL, have near identical problem statement. We find that this could be attributed to the fact that on the Codeforces platform, harder problems from easy contest (div2) could appear also in harder contest (div1) as easier problems. We show an example below, in which in training set the problem URL is [https://codeforces.com/problemset/problem/841/C](https://codeforces.com/problemset/problem/841/C) and in test set it is [https://codeforces.com/problemset/problem/840/A](https://codeforces.com/problemset/problem/840/A).

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. Diversity in Rejection Sampling Fine-Tuning: The paper fine-tunes the model on correctly generated trajectories. Given the potential for this approach to bias the model towards successful solutions, would the authors consider including near-correct or diverse trajectories to introduce a broader range of reasoning paths? This could enhance the model’s adaptability to different problem-solving strategies.
2. Adaptive Prompt Selection for Reducing Computational Requirements: The extensive grid search across prompt types and model configurations is resource-intensive. Have the authors explored adaptive or meta-learning techniques to streamline prompt selection, reducing computational demands while still achieving optimal prompt configurations? Such techniques could make the framework more accessible for settings with limited computational resources.
3. Detailed Analysis of Prompt Failure Modes: The paper mentions that certain CoT prompt configurations may degrade performance, particularly in multi-turn settings with higher sample budgets. Could the authors provide a more granular analysis of these failure modes? Understanding specific prompt combinations or scenarios where performance degrades could guide practitioners in avoiding ineffective configurations.
4. Balancing Exploration and Exploitation in Feedback Prompts: The study notes that finer-grained feedback prompts may encourage exploitative behavior in models, potentially limiting performance on complex tasks. Have the authors considered any strategies for adjusting the feedback granularity dynamically based on task difficulty or performance? This could help achieve a balance between exploration and exploitation for more challenging problems.
5. Transferability of the Findings to Other LLM Architectures: The paper experiments primarily with the Llama series and GPT-4o models. To what extent do the authors believe that the findings, especially related to prompt configurations and RFT, would transfer to other LLM architectures with different pretraining characteristics? For instance, would models with fewer parameters or more domain-specific training exhibit similar responses to the same prompting strategies?
6. Future Directions for Enhancing Multi-Turn Reasoning in LLMs: Given the current results, what specific directions do the authors see as most promising for advancing multi-turn reasoning capabilities? Are there any particular technical approaches, benchmarks, or modifications to current LLM architectures that the authors believe could improve multi-turn performance on reasoning-heavy tasks?
soundness: 4
strengths: **Originality**:  
This work provides a unique contribution by systematically exploring multi-turn code generation for large language models (LLMs) through diverse prompt configurations, including **chain-of-thought (CoT)**, **reasoning prompts**, and **execution feedback**. While prior work has explored CoT and single-turn code generation, this paper’s emphasis on **multi-turn prompting in competitive programming benchmarks** addresses an under-explored area. The study’s integration of multi-turn CoT with finely tuned execution feedback as a **comprehensive prompting framework** offers an original approach to improve model reasoning in multi-turn contexts. Additionally, the use of **Rejection Sampling Fine-tuning (RFT)** to internalize CoT reasoning across turns is a creative and impactful extension, enabling models to generalize reasoning capabilities without prompting at inference time.

**Quality**:  
The paper demonstrates high quality in its rigorous experimental design and breadth of evaluations. Through an extensive **grid search** of prompt combinations, the authors provide a robust dataset for comparing single-turn and multi-turn settings across a **variety of model architectures and sizes** (Llama 3.0/3.1, GPT-4o). The benchmarks, CodeContests and TACO, are well-chosen for evaluating competitive coding capabilities, allowing the paper to contribute valuable insights for LLM applications in algorithmic and complex reasoning tasks. Methodologically, the work includes detailed ablations, sensitivity analyses, and cross-model comparisons, ensuring that the findings are well-supported and generalizable.

**Clarity**:
The paper is well-structured, clearly outlining the motivation, objectives, and experimental setup. The **framework for prompt deconstruction** is presented systematically, with visual aids like **Figure 1** that clarify the operational structure of single- and multi-turn code generation. Each type of prompt (reasoning, instruction, feedback) is explained with appropriate examples, making the framework comprehensible for readers. The authors’ discussion on **exploration-exploitation dynamics** and performance limitations in multi-turn settings enhances clarity by contextualizing how feedback granularity impacts model behavior. Additionally, the conclusions are tied back to the primary research questions, reinforcing the relevance of the findings.

**Significance**:  
The study makes significant strides in understanding and advancing multi-turn reasoning for code generation in LLMs. By introducing a unified prompting framework and evaluating its efficacy in algorithmically demanding benchmarks, the paper sets a strong foundation for future research in **multi-turn, reasoning-heavy applications** for LLMs. The RFT approach has implications beyond competitive programming, as it suggests that **prompted reasoning behaviors can be internalized** within the model, potentially reducing computational costs for other complex, multi-step tasks. Given the trend toward **LLM-based agents** capable of autonomous decision-making, this paper’s insights into adaptive CoT, feedback prompting, and exploration-exploitation trade-offs are timely and have high applicability.

**Conclusion**:
In summary, this paper’s originality in multi-turn prompting, its methodological rigor, clarity, and significance make it a valuable contribution to the field of LLM code generation. The findings not only enhance understanding of CoT in multi-turn settings but also provide practical strategies for optimizing reasoning in LLMs, making this work highly relevant for both researchers and practitioners.
summary: The paper investigates the effects of varied prompting techniques, including chain-of-thought (CoT), reasoning, and execution feedback prompts, on the performance of large language models (LLMs) in multi-turn code generation. It focuses on understanding the mechanics and efficacy of these prompts on complex programming tasks using benchmarks like CodeContests and TACO across models of different sizes (e.g., Llama 3.0/3.1, GPT-4o). Unlike simpler code generation tasks, competitive coding demands algorithmic reasoning and has not been widely explored with unified evaluation metrics for multi-turn code generation. 

Key contributions include:

1. **Framework for Multi-Turn Code Generation**: The paper establishes a unified framework that enables a mix of single-turn and multi-turn prompting techniques, incorporating reasoning, instruction, and feedback prompts. This approach aims to build robust baselines for multi-turn performance and to measure trade-offs between sampling budgets and model capabilities.

2. **Experimental Insights**: The authors’ extensive grid search reveals that combining reasoning and instruction prompts significantly enhances single-turn model performance, especially for larger models and more difficult tasks. However, for multi-turn settings, while combining with CoT prompts boosts performance, using multi-turn alone has limited benefits and can even reduce performance at higher sample budgets due to less diverse outputs.

3. **Internalized CoT**: The study adapts the RFT, which allows models to internalize multi-turn CoT reasoning during training. RFT-trained models demonstrate improved performance and reasoning abilities in the absence of CoT prompts at inference, showing promise for scaling up multi-turn code generation capabilities.

4. **Exploration-Exploitation Dynamics**: The analysis indicates that detailed feedback can induce exploitative behavior in models, which is beneficial for simpler problems but counterproductive for complex ones where diversity in code generation is critical. This highlights the need for feedback strategies that balance between exploiting known solutions and exploring new ones, particularly in high-difficulty settings.

In conclusion, this work underscores that multi-turn code generation is not inherently beneficial without adaptive strategies like CoT prompts and RFT. The findings advocate for tailored prompting configurations based on problem difficulty and suggest that further advancements in multi-turn code generation could benefit real-world applications such as repository-level LLM agents capable of reasoning, self-repair, and complex interaction with environments.
weaknesses: While the paper makes significant contributions to the understanding of multi-turn code generation in LLMs, there are a few areas where improvements could be made to strengthen the study further.

**1. Limited Scope of Multi-Turn Framework**  
The paper’s framework, while well-designed for competitive programming benchmarks, focuses primarily on **chain-style multi-turn code generation**. This approach limits the exploration of more complex **tree-structured trajectories** or **backtracking mechanisms** that could better emulate real-world multi-turn interactions in agent-based LLM applications. Exploring additional trajectory structures—such as decision-tree-style paths where LLMs could backtrack or revise earlier steps—would add depth to the study and provide insights into more adaptive reasoning scenarios. Extending the framework to incorporate these structures could lead to a more comprehensive understanding of multi-turn reasoning dynamics.

**2. Potential Bias in Rejection Sampling Fine-Tuning (RFT)**  
Although Rejection Sampling Fine-Tuning (RFT) is a compelling method for embedding reasoning behavior, the paper’s reliance on **correctly generated trajectories** may introduce a bias toward successful patterns, potentially limiting the model’s adaptability to a broader range of problem-solving approaches. By focusing only on “successful” reasoning patterns, the model may overfit to specific prompt types or problem formats. Future work could consider **diversifying the fine-tuning data** to include not only correct but also nearly correct attempts with varied prompts, allowing models to learn from a broader array of problem-solving paths.

**3. Dependence on High-Resource Settings**  
The paper’s approach requires considerable **computational resources** due to the grid search over a large set of prompt combinations and the use of multi-turn, CoT, and feedback prompts across different model sizes. This may limit the practical application of the findings for research groups or practitioners with restricted compute budgets. Introducing methods for **adaptive prompt selection** that can identify effective prompt configurations more efficiently (e.g., meta-learning approaches) could make the approach more accessible and scalable, especially for real-world applications where computational resources may be constrained.

**4. Limited Discussion of Prompt Failure Modes**  
The paper could benefit from a more detailed **analysis of prompt failure modes**, particularly in cases where certain prompt combinations degrade performance in multi-turn settings. While the paper briefly notes that CoT may harm performance by obstructing LLM context or encouraging suboptimal plans, it lacks a thorough exploration of specific cases where multi-turn CoT and feedback combinations lead to failure. A more in-depth examination of these failure patterns, especially for challenging tasks, would provide actionable insights into optimizing prompt configurations and mitigating these issues in future iterations.

**Conclusion**  
In summary, while the paper delivers strong contributions to multi-turn prompting strategies, expanding the framework’s scope to include more diverse trajectory structures, and refining RFT to include a wider range of training data. Additionally, addressing the high computational costs and analyzing prompt failure modes more deeply would make the work more robust and accessible for practical application. These improvements could further strengthen the study and support its stated goal of advancing multi-turn reasoning capabilities in LLMs.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Please answer the questions in the weakness section.
soundness: 2
strengths: -	The paper brings together the evaluation of a wide range of prompting strategies in both single turn and multi-turn for multiple models on a challenging code dataset.
-	Extensive set of experiments with an extensive grid search
-	Interesting idea of splitting the CoT prompts into reasoning, instruction, and feedback so that we can understand what strategies work best for different model sizes and problem difficulties. 
-	Multi-turn code generation findings reinforce insights from previous papers --- just multi-turn is not sufficient but combining with CoT might improve performance. 
-	Fine-tuning on multi-turn CoT data is a novel approach that shows promising results.
summary: This paper evaluates various prompting strategies in single-turn and multi-turn code generation settings. For single-turn generation, it explores chain-of-thought (CoT) prompting, dividing it into reasoning prompts and instruction prompts. In multi-turn generation, the paper investigates multiple error feedback strategies. The evaluation covers the CodeContests and TACO benchmarks, focusing on Llama models and GPT-4o. The authors also fine-tune the models on multi-turn CoT data to internalize reasoning within the LLMs.
weaknesses: - **Limited novelty**: There is substantial existing research on CoT and self-repair in the code domain, which somewhat limits the paper's originality.

- COT-Retry prompts seem like an arbitrary combination of prompts. Why this particular combination?
The paper lacks any chain of thought prompting specific to reasoning about why the test failed or how to fix the test.


- For additive reasoning prompts experiment in table 6, what happens if each sample uses a different COT prompt? 


Some of the details are missing/not clear that gives me less confidence on the results:

1.	**Baseline for Fine-Tuning**: There should be a baseline that fine-tunes on problem-solution pairs (without CoT data). This would help clarify whether RFT fine-tuning offers benefits beyond simple fine-tuning.

2.	**Clarity on CoT and Multi-Turn CoT**: Tables 1 and 2 could more clearly explain CoT and multi-turn CoT strategies. Are they different across models? Are they also different across sample sizes?

3.	**Another baseline for table 8**: How does table 8 results compare to just resampling from the base model? Is feedback even playing a role in these multi-turn settings? 

4.	**Explain Figure 6**: Why does the multi-turn CoT differ from the base model at max_turns=1? Shouldn’t they be identical if CoT-Retry doesn’t apply CoT in the first turn?

5.	**Misinterpretation of results**: Some results are misinterpreted in the text. For instance, Figure 4 uses different axes across charts, making it difficult to interpret trends accurately. . Also, we cannot  say that 1% to 2% is better than 30% to 40% or other way around.

---

confidence: 3
contribution: 2
initial_rating: 8
presentation: 3
questions: 1.How well do the findings generalize to other types of code generation tasks beyond competitive programming? Are there more complicated benchmarks available to evaluate?

2.How large finetuning dataset is required to train a sufficient model that internalized the CoT? Need more finetuning details.

3.How does the finetuned model perform on unseen code generation tasks?
soundness: 4
strengths: 1.Comprehensive evaluation of CoT prompting strategies across different model sizes and benchmarks. I found the negative results that Reasoning prompts not additive interesting.

2.Usage of pass n@k metric for fairer comparison between single-turn and multi-turn approaches.

3.Thorough analysis of the impact of different error-feedback granularities.

4.Investigation of fine-tuning on multi-turn CoT data and its effects on model behavior, showing effectiveness to internalize CoT steps for LLMs.
summary: This paper investigates the effects of various prompting strategies on large language models (LLMs) for code generation tasks, focusing on multi-turn interactions and computational requirements. The authors conduct a comprehensive study using competitive programming benchmarks (CodeContests and TACO) across multiple LLM families and sizes. They explore single-turn vs. multi-turn code generation, chain-of-thought (CoT) prompting techniques, execution feedback in multi-turn settings, and the impact of model size and problem difficulty on prompting effectiveness. The study also examines fine-tuning models on multi-turn CoT data.
weaknesses: 1.Although this work has solid empirical evaluations and good presentations, the novelty is a bit limited by combining existing works together, including CoT strategies, error-feedback granularities, and Pass n@k. The core contribution is primarily a prompt engineering effort combined with the evaluation of existing LLM capabilities in code generation. While valuable, this doesn't represent a significant theoretical or methodological advancement in the field. As for the RFT part, although I think it is a novel method used here, I would like to see more details on it to explain it clearly, with details on how it is finetuned with.

2.Limited theoretical analysis of why certain prompting strategies work better than others. Although the authors presented solid empirical results on the prompting strategies ablation, I will still wonder why it works. Plus, since there is no best CoT for every case, i still think cases can be categorized into a bunch of properties, would it be better to have a task analysis model first, to decide which CoT to be applied based on these properties? I would be happy to see the answer to it.

3.Potential overfitting concerns with fine-tuning on multi-turn CoT data generated from the same benchmark used for evaluation, since currently the training process is not clearly described, i keep this as a doubt and look forward to the authors' response.

4.Figure presentations: I found some figures in the appendix not using vector figures, which are in low-resolution, starting from line 1100 and after. And some figures don't have proper captions, starting from line 1524 and after. The typesetting in the appendix should also be reorganized to avoid large blank pages. In a word, I think the good presentation in the main text should be kept as well in the appendix for a good ICLR paper.

Meta-review: This paper performs a systematic evaluation of the interplay between chain-of-thought reasoning and multi-turn generation in LLM code generation. The question of the best strategies for code generation (especially single-turn vs. multi-turn) has received significant recent interest. They identify several high-level strategies that are effective across a range of models. There are some lingering concerns about both the clarity of the writing (especially with respect to the array of different approaches and experiments considered), as well as the novelty. I especially agree that currently, the key takeaways are somewhat obfuscated (for instance, they are nowhere to be found in the title or abstract). Thus, while the reviews lean towards acceptance, I strongly encourage the authors to improve the clarity and positioning of their paper.

Decision: Accept (Poster)