ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments
Authors: Hojae Han
seung-won hwang
Rajhans Samdani
Yuxiong He
Venue: ICLR 2025 Poster (2025)
Keywords: Large language models
Multi-turn code generation
Benchmark
Primary area: datasets and benchmarks
TL;DR: A novel multi-turn-based code generation benchmark with diverse feedback combinations

Abstract: Large language models (LLMs) have proven invaluable for code generation, particularly in interactive settings. However, existing code generation benchmarks fail to capture the diverse feedback encountered in multi-turn interactions, limiting our ability to evaluate LLMs in these contexts. To address this gap, we present a set of novel benchmarks that explicitly model the quality of feedback provided to code generation LLMs. Our contributions are threefold: **First**, we introduce CONVCODEWORLD, a novel and reproducible environment for benchmarking interactive code generation. CONVCODEWORLD simulates 9 distinct interactive code generation scenarios while systematically combining three types of feedback: (a) compilation feedback; (b) execution feedback with varying test coverage; (c) verbal feedback generated by GPT-4o with different levels of expertise. **Second**, we introduce CONVCODEBENCH, a fast, static version of benchmark that uses pre-generated feedback logs, eliminating the need for costly dynamic verbal feedback generation while maintaining
strong Spearman’s rank correlations (0.82 to 0.99) with CONVCODEWORLD. **Third**, extensive evaluations of both closed-source and open-source LLMs including R1-Distill on CONVCODEWORLD reveal key insights: (a) LLM performance varies significantly based on the feedback provided; (b) Weaker LLMs, with sufficient feedback, can outperform single-turn results of state-of-the-art LLMs without feedback; (c) Training on a specific feedback combination can limit an LLM’s ability to utilize unseen combinations; (d) LLMs solve problems in fewer turns (high MRR) may not solve as many problems overall (high Recall), and vice versa. All implementations and benchmarks will be made publicly available at https://huggingface.co/spaces/ConvCodeWorld/ConvCodeWorld

Published as a conference paper at ICLR 2025
CONVCODEWORLD: BENCHMARKING CONVERSATIONAL CODE GENERATION IN REPRODUCIBLE FEEDBACK ENVIRONMENTS
Hojae Han Seung-won Hwang Rajhans Samdani Yuxiong He*
Snowflake Al Research SSouul National University
ABSTRACT
Large language models (LLMs) have proven invaluable for code generation, particularly in interactive settings. However, existing code generation bench- marks fail capture the diverse feedback encountered in multi-turn interac- tions, limiting our ability to evaluate LLMs in these contexts. To address this gap, we present a set of novel benchmarks that explicitly model the quality of feedback provided to code generation LLMs. Our contributions are three- a novel and reproducible environ-
fold: First, we introduce CONyCODEWORLL ment for benchmarking interactive code generation. CONYCODEWORL,, simu- lates 9 distinct interactive code generation scenarios while systematically com- bining three types of feedback: (a) compilation feedback; (b) execution feed- back with varying test coverage; (c) verbal feedback generated by GPT-4o with different levels of expertise. Second, we introduce CONYCODEBENCH static version of benchmark that uses pre-generated feedback logs, eliminat- ing the need for costly dynamic verbal feedback generation while maintaining strong Spearman's 's rank correlations (0.82 to 0.99) with CONvCODE WORLD Third, extensive evaluations of both closed-source and open-source LLMs in- cluding RI-Distill on CONvCoDEWORLD reveal key insights: (a) LLM perfor- mance varies significantly based on the feedback provided: (b) Weaker LLMs. with sufficient feedback, can outperform single-turn results of state-of-the-art LLMS without feedback; (c) Training on a specific feedback combination can limit LLM's ability to utilize unseen combinations; (d) LLMs solve problems in fewer turns (high MRR) may not solve as many problems overall (high Re- call), and vice versa. All implementations and benchmarks are publicly available at
fast,
co/
INTRODUCTION
Human-Al pair programming has become promising approach to boost software development pro- ductivity, where large language models (LLMMs) iteratively refine the code from developers' feed- back. However, most existing benchmarks focus single-turn scenarios, where LLMs are expected
to generate executable code in one attempt Chen et al. 2021]: Hendrycks al. (022)) Austin et al.
(2021) Li et al. (2022); Zhuo et al. 2024).
To address these gaps, we introduce CONVCODEWORL.D { left panel in Figure I. novel envi- ronment for benchmarking interactive multi-turn code generation across diverse feedback combina- tions. CONyCODE CODEWORd features nine scenarios by combining three feedback types: (a) compila- tion feedback, (b) execution feedback with partial and full test coverage, and (c) novice and expert level verbal human feedback. We simulate human feedback using GPT-4o (OpenAl (024) to gener- ate verbal responses, ensuring reproducibility and cost-efficiency at only 1.5% of the cost of human annotation (Appendix C.2).
While replacing expensive human intervention with L.MSs in WORNOL reduces costs, it can still be expensive due to computational overhead or API fees, and latency due to LLM response.
Work done while visiting Snowflake AI Research. Correspond seungwonh @snu.ac.kr

---

Published as a conference paper at ICLR 2025
ConvCode
ConvCode
WORK WORLD
BENCH
A Reproducible Environment with Diverse Feedback Combinations
Cost-Effective Benchmark Strongly Correlates CONVCODEWORLD
Feedback
Log Data
Prev. code & Feedback
Feedback
Code
LLM
Env:
Feedback Collection
LLM
Code
ref.
Code
Compilation Feedbarck
Execution on fercback (Test Coverage: Full
Partiel)
Spearman Rank Correlation: 0.95
Verbal Feedbbakk
Novice
Expert)
0.650
ConcCadsBench MRR
Feedbbckk:
0.602
Focdback
0.575
Foodbacck:
MRR
025
GPT-4c (525)
GPT-4 (TU3)
0.500
0.475
GPT-4-Tuto 154.8
GPT-4-Turbo 163
0.45
0.500
ConVCodeWarld MRR
Figure (Left) CONyCODEWORLD is dynamic, reprodueible environment that simulates nine distinct feedback scenarios by combining three types of feedback. (Right) CONyCODEBENCH is a static version of the benchmark that uses pre-generated logs and strongly correlates with CON- VCODEWORLD. Together, these frameworks provide comprehensive, cost-effective approach for evaluating LLMs in multi-turn, feedback-driven code generation, enabling scalable and consistent benchmarking across diverse feedback combinations.
To address these issues, we introduce CONYCODEBENCH $44 right panel in Figure 1 static benchmark using pre-generated feedback logs. CONYCODEBENCH eliminates the need for real- time feedback generation while maintaining strong correlation with CONVCODEWORL. (Spear- man's rank 0.82-0.99; $5.3), offering a cost-effective and scalable solution for large-scale LLM benchmarking.
Existing benchmarks like InterCode (Ynng et al. 2023) and MINT (Wang et al. 2024) lack the variety feedback combinations needed for comprehensive LLM performance assessment ($2). Ad- ditionally, their reliance on LLM calls for verbal feedback increases costs. Our study stands out by (a) offering reproducible environment with 9 unique feedback combinations, and (b) providing cost-effective benchmark using pre-generated logs, avoiding costly LLM calls for verbal feedback while maintaining strong correlation with live results.
Through extensive experiments using both CONYCODEWORLD and CONYCODEBENCH across 21 different open and closed-source models including R1-Distill (DeepSeek-Al al. (025); Ap- pendix A), we have gathered several key insights:
Feedback Combinations Diversifying Evaluation: LLM performance varies across feedback settings, with feedback combinations affecting model rankings, highlighting the need for evalu- ation across diverse scenarios.
Weaker LLMs, with suffi- cient multi-tumn feedback, can surpass -fate-of-the-art models in single-turn scenarios without feedback. This emphasizes the importance of interactive multi-turn code generation.
Weaker Models with Feedback Surpassing Single-Turn SOTA:
Generalization Challenges: Models trained on limited feedback struggle to generalize to un- seen combinations, highlighting the difficulty of adapting LLMs to new scenarios.
MRR and Recall Trade-off: LLMs that efficiently solve problems in fewer turns (high MRR) may not solve as many problems in total (high Recall), highlighting a trade-off between effi- ciency and problem coverage.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
Code generation benchmarks have traditionally focused on single-turn generation from natural lan- guage problem descriptions (Chen al., 2021; Austtin al. 2021 Li et al.] 2022 Zhuo et al. 2024). More recently, LLM performance has improved through interactions with external tools, such interpreters for compiling, executing test cases, and verbal feedback, resulting in more
curate outputs (Shinn et al. 2023 Madaan et al., 2024] Chen et al.)
2024 Olausson et al. 2024
This shift has led the development of multi-turn benchmarks like InterCode (Yang et al. 2023 and MINT (Wang et al. 2024).
However, existing multi-turn benchmarks re- main limited in feedback diversity. InterCode focuses on compilation and partial execution feedback but lacks full test coverage and ver- bal feedback. MINT generates verbal feedback via GPT-4, reducing human-in- in- nn-thhellopp evalua- tion costs, but its feedback scope is narrow and requires costly LLM calls for each evaluation.
Table I: Feedback combinations (Ω;; 3.2 across InterCode, MINT and CONYCODE structed by different feedback types ($3.).
InterCode
CONYCODEWORLD
$o.0.00 fus 1)
/
fer
Our study presents (a) CONVCODE WORLD, reproducible environment with nine unique feedback combinations (Table D. and (b)
fes
(e.
X
V
X
fe. f.)
(ee 1
X
CONYCODEBENCI,, a cost-effective bench- mark that maintains high correlation with live environment by using pre-genn rre-generated logs, elim- inating the need for costly LLM calls to provide verbal feedback. We further discuss the distinction of CONVCODEWORDD WOLL in Appendix B
fo:
X
(e.
X
X
CONYCODEWORLD. REPRODUCIBLE FEEDBACK ENVIRONMENTS
In real-world settings of interactive code generation, the types and combinations of feedback can vary signifi- cantly due factors such the availability of feed- back from code execution (e.g., eгror messages, output) and the expertise of the feedback provider. These varia- tions, particularly the provider's expertise, can strongly influence the quality of the verbal feedback when offered.
Table 2:
By providing diverse feedback types, with different coverage levels in execution and natural language feedback, encompasses a broader range of alistic scenarios. indicates partial cov- erage ($3.1) with specific limitations: Syntax errors only, "Limited by test cov- erage feedback provider, and Potential misguidance due to limited expertise.
To effectively evaluate LLMs under these varying con- ditions, we propose CONyCODEWORD reproducible environment designed to simulate wide range of interactive code generation scenarios. Two key features of CONvCODEWORLD are as follows: (a) Encompassing Diverse
novel and
Fault Localization
Guidance Refivement
Feedback
fe
A
X
Scenarios: CON- ORR.D covers nine distinct feedback combi- nations that occur in practical settings; (b) Ensure the Reproducibility of Evaluation: CONYCODEWORLD provides a consistent and repeatable framework for as- sessing the performance of LLMs.
fe
4
X
X
fu
4
3
3.1 FEEDBACK CATEGORIZATION
To accurately simulate real-world feedback interactive code generation, we focus on two critical components: (a) Fault Localization: Identifying the specific parts of the code where issues errors occur; (b) Guidance for Refinement: Offering suggestions or instructions on how to correct the identified issues.
As means of obtaining such information, we consider three different types of feedback: compilation feedback, execution feedhack, and verbal feedback.
3

---

Published as a conference paper at ICLR 2025
Compilation Feedback (fe) Originated from the compiler, this feedback identifies syntax and type-checking errors but cannot localize logical or runtime errors. As a result, Table marks this with A for partial fault localization. Additionally, compilation errors do not offer refinement guidance.
Execution Feedback Derived from code execution, this feedback includes runtime errors and test run results. Full or partial fault localization is provided, depending on test coverage (TC): (a) Full TC (f²): Inspired by Test-Driven Development (TDD: (TDD::Beck] 2022), complete test cases allow precise fault localization, identifying where and under what conditions the code fails. This provides details on the failure's 's location and triggering inputs; (b) Partial TC (f.): In more realistic settings with partial test coverage, fault localization is limited to tested code lines, potentially leaving faults in untested sections undetected. This type of feedback simulates incomplete real-world test suites, where only subset of possible execution paths is covered. Refinement guidance is not provided in either full or partial test coverage executions.
Verbal Feedback Verbal feedback in our benchmark is generated by LLMs simulating human feedback, ranging from novice to expert levels. This feedback could emulate responses from hu- mans, such as experts guiding I.LMs to generate code, or novices without coding expertise. Both fault localization and refinement guidance are provided verbally,, but the extent and accuracy of this feedback depend on the simulated provider: (a) Novice-Leve (f.): At this level, the LLM sim- ulates novice feedback, which tends to rely heavily on other feedback types (e.g., compilation or execution feedback) and often restates observed errors without deeper understanding. Refinement guidance may be incorrect or absent, due to the novice's limited expertise simulated by the LLM's potential hallucinations. (b) Expert-Level (f²):: Expert feedback reflects scenarios where expert programmers use I.LMs to automate simpler tasks, allowing them to concentrate on more complex coding challenges. This feedback is simulated by the LLM to provide detailed fault localization and code refinement guidance. It generates the feedback an expert programmer might give, focusing on resolving issues with deep understanding of programming concepts and the expected functionality.
3.1.1 VERBAL FEEDBACK GENERATION
We generate fo and f* by GPT-4o with in-context learning (Dong et al., 2022). We chose GPT-4o as we found it to be best at following instructions and minimizing risks such as ground truth code leakage, as discussed in Appendix C.5
Generation of fe: Novice-level verbal feedback is constructed by verbalizing outputs from com- pilation and/or execution feedback, possibly supplemented with language model predictions.
Generation of f:: Expert-level verbal feedback is produced by showing the agent's 's code with the correct reference code (Wang et al. 2024), enabling comparison and subsequent detailed feedback on required modifications. We perform extensive analysis to ensure no ground truth code leaked during f generation (see Appendix
C.5 for analysis on this).
See Appendices C.6 for comparative analysis of verbal feedback using different LLMs, [ for the in-context examples, and [oor a generated example of ft. For the detailed prompting methods for the fo and f construction, please refer our codebase
Reproducibility and Cost-Efficiency Compared to Human Annotation Manual annotation of verbal feedback is costly and lacks reproducibility. Instead. we use GPT-4o, as supported by prior studies demonstrating the effectiveness of LLM -LM-gnnerated feedback in benchmarks (Wang et al.) 2024 Yao al., 2024). This approach improves reproducibility by using consistent feedback provider and reduces annotation costs to about 1.5% (Appendix C.2) of those for human annotators.
3.2
FREDBACK COMBINATIONS
In each of our turns, we simulate different world interactive code generation scenarios by com- bining representative feedback combinations. We represent feedback settings by taking Cartesian product across compilation feedback settings, execution feedback settings, and verbal feedback set- tings. In particular, formalize feedback combination ? as tuple of feedback expressed by

---

Published as a conference paper at ICLR 2025
regular expression notation:
The choices of feedback settings is simply dictated by these observations: (a) Compilation feedback fe is always present since cheap and universally available; (b) Execution feedback varies among being unavailable (ő), available with partial test coverage (fe), or with full test coverage (f²); (c) Verbal feedback can also unavailable (Φ), available with novice-level (f.). or with expert-level (1²), By combining these options-1 for compilation feedback, 3 for execution feedback, and 3 for verbal feedback-we obtain distinct feedback combinations.
Each feedback combination Ω reflects unique real-world scenario, allowing us comprehensively evaluate LLMs under diverse conditions as listed in Table []
Now it is easy formalize the interactive code generation in CONYCoDE For each turn t, the target code generation model M iteratively generates the next version of code C/41 from the problem description z, the generated code C,M, and the corresponding tuple of feedback Ω4;
C,41 = MMxx C,,
Ste)
CONVCODEBENCH: A STATIC BENCHMARK FOR EFFICIENT EVALUATION
While CONYCODEWORLD provides comprehensive live benchmark for evaluating LLMs in- teractive code generation scenarios, it requires access to an LLM for verbal feedback generation. Although this approach is more efficient and reproducible than using human annotators, it still intro- duces additional overhead, cost, and potential reproducibility issues, especially when using closed API models like GPT-4o. To address these challenges, we propose CONVCODEBENCH, static benchmark designed to complement MAANCODEWWOLL.
CONYCODEBENCH leverages feedback logs generated by fixed reference model interacting with GPT-4o. The benchmark presents pre-generated irreersationninclddini the code produced by the reference model and the corresponding feedback, such as verbal feedback by GPT-4o-and tasks the target code model with refining the code. We revise Equation2tte formalize CONYCODEBENCH as follows. For each turn f, the target code generation model M is provided generated code C.M from a reference model M, and the corresponding tuple of feedback Ω provided to outputs generated by M. Given the model and feedback corresponding reference model, the target model M generates the next version of code C.A1
CM M(I; CMM
(3)
This approach offers several advantages:
Elimination of Dependency on External LLMs or APIs for Verbal Feedback Generation:
By using static feedback logs, CONVCODEBENCH reduces costs and latency associated with real-time LLM interactions.
Parallel Processing of Inference Calls: The static nature of the benchmark allows for batched evaluation requests across all turns, enabling faster turnaround times.
Enhanced Reproducibility: Utilizing fixed logs ensures consistent evaluations, further increas- ing reproducibility.
One key concern when using CONVCODEBENCH is the bias introduced by pre-generated interaction logs prompting the question: Can we ensure high correlation between static and live benchmarks by an appropriate choice of reference model?
Table 3: Performance of three different LLMs turn (i.e. the initial code generation with- out feedback) and at turn 10 on CONVCODE- WORLD where Ω (fes d. f.).
We hypothesize that using logs from weaker model, where the generated code still requires re- finement after multiple turns, allows for better differentiation among models based on their ability to improve unsolved code.
Pass 1
Model
Turn
Turn
Code B--
21.8
55.2
DeepSokk-Cder-6 Boodder-.7B--aaae In-ttutt GPT-4-0633
35.2
83.1
46.0
92.5
5

---

Published as a conference paper at ICLR 2025
Based on this rationale, used CodeLlama-7B- Instruct as a reference model, as is worse than many other models at both turns 0 and 10 (see Table We find that creating CONVCODEBENCH with this model yields very strong correlations with live settings. When comparing models on CONVCODE WORLD and CONyCoDEBENCH, we obtained Spearman's rank correlations between 0.82 and 0.99. We find that using CodeLLlama-71 Instruct as the base model outperforms both DeepSeck-Coders 7B IB-Instruct (a stronger code model) and GPT-4 (one of the state-of-the-arts) reference models ($5.3).
In summary, we find that CONYCODEBENCI is a great way of comparing code models within the framework of CONvCODEW WoDeeWoLDD despite relying on logs from reference model because of strong rank correlations across the two setups.
EXPERIMENTS
Using CONYCODEWORK and CONYCODEBENCH, we conduct comprchensive experiments to evaluate LLMs' interactive code generation capabilities across diverse feedback combinations. This section outlines experimental setup ($5.I). results on CONYCODEWORLD ($5.2), and results on CONNCODEBENCI (5.3)
5.1
SETUP
To implement CONVCODE WENVCODEWORLDD WORLD, we extended BeseoddBencc-Full-nnnseee (Zhuo et al. 2024). a single-turn Python code generation benchmark, into interactive framework using a custom prompt pipeline built using DSPy (Khattab et al. 2024) (see Appendix E for the implementation details). BigCodeBench was selected for three key reasons: (a) its highly challenging problem sets (as of the writing of this paper, the highest performance on this data is 51.1% of Pass @ 1); (b) its large scale, with 1,140 problems, offering higher generalizability than smaller benchmarks like HumanEval (Chen et al., 2021] 164 problems) and MBPP-sanitized (Austin et al., 2021] 399-427 problems); and (c) comprehensive test coverage an average of 5.6 cases per problem with 99% branch coverage envarag--naallngg the evaluation of wide spectrum of execution feedback scenarios, ranging from partial to full test coverage.
Evaluation Metrics In the interactive scenario, where code is iteratively refined based on feed- back, we focus on two aspects for evaluation: (a) the number of turns it takes to produce correct code, with fewer turns being preferable, and (b) whether the model can eventually solve the problem within set number of turns In our experiments, we set 10.
To capture these aspects, we use Pass@1 (Chen et al. 2021) as the core metric to assess code correctness at each turn and adapt two complementary metrics from information retrieval: (a) Mean Reciprocal Rank (MRR): & where the turn at which the model produces correct code. If no correct code is generated within turns, the score is set to O; (b) Recall: I if the model produces correct code within n turns.
Baseline LLMs We extensively evaluated closed-source and open-source LLMSs rang- ing from 7B to 70B (a) Closed-Source: We select three OpenAl L.LMs GPT-4-0613. GPT-
--Turbo-2024-04-00 and GPT-44-0024-05-11; (b) Open-Source: -lama-3.1-70B-Inae
(Dubey (Zhu et al. 2024); MoE
cet al. 2024),
CeepSecck-Code--2--Lee
model; total params: 16B; active params: 2.4B), DeepSeek-Coder-33BB Coder-3dB-rr-BB- 3inet333---nntrrtttt
Instruct (Guo et al. 2024),
RefiectionCoder-DS--- (Ren et al.
2024). Reflection
DS-6.7B, Qwen S.renl.5--2B-CC1 (Bai et al.] (023). Qwenl.3-32B Chat. CodeQwenl.55 Chat,, MarCooder2-15B-Instnnu Instruct, and CodellJama-7B--insttne In Appendix [ we further included the results of two recent
(Roziere al. 2023 [1
Codel 13BB-
RI-Distill (DeepSeck-Al al. 2025) models and their base models on CONyCoDEWORLA
2 While we attempted smaller models like D.epScck it failed to follow interactive code generation format, resulting degeneration.
ht //huggi angfacce ce. co/obi gcode/ setood//tt.n htt///tarrooeeee 115--155---- 1 nst ruct-vv0 We excluded Jama-708-Bl due to its 4K token length limitation, which is too short for interac- tive code generation.

---

Published as a conference paper at ICLR 2025
Table MRR results on CONYCODEWORLD. X indicates that no feedback of that type is provided (é). The leftmost results, with three X, represent f2 (Ø, Φ, Φ). corresponding to single-turn code generation without any feedback. For each column, bold and underscore indicate 1st and 2nd place performance within the same model group.
Complation Feedback
fc
f0
Jc
fc
Je
Jc
fc
Execution Feedback
fx
Verbal Feedback
Closed Source Models
GPT-4-0613
52.1
63.1
GPT-4-Turbo-202444--09
48.0
48.0
51.8
54,8
48.0
52.6
56.4
62.4
64.3
64.5
GPP-40-2024405-13
50.8
50.8
55,0
57.9
50.8
55.1
58.6
63.3
64.7
65,3
Open-Source Models (3= 338)
45.4
45.4
49,9
53.4
45.4
50.8
55.2
60.7
62.6
63.3
DeepSeek-Coder-338--
43.4
43.6
45.5
48.0
58.5
ReflectionCoder-DSS
41.6
41.6
42.9
42.9
41.6
45.6
48.1
57.7
58.2
58.9
Qwent @wenl_5-72B8 5-221--77---ha
32.9
33.0
35.8
38.3
33.0
38.6
41.4
50.6
52.0
52.7
5.enl.5.32B Chat
32.0
32.0
35.3
36.7
32.0
39.7
47.4
42.6
40.8
Codel 3-B---4---r-rr
28.8
28.8
31.0
31,9
28.8
32.5
35.1
48.7
49.2
49.8
Open
Ilama-3. 3.1n--3.-88B Instruct
Insttrctttt
31.4
31.5
34.0
34.6
31.5
36.
39.1
49.4
49.8
51.3
DeepSeek Coder-VT -2---22-- V2-Lite Instruct
38.3
38.3
40.5
41.7
38.3
42.0
43.8
52.7
$2.9
53.3
DeepSeek- Cader-6.7P Ins-tutt
35.2
35.2
36.2
36.1
35.2
40.5
53.3
53.2
53.9
CefeetionCode:-DS--.. 7B CodeQwen1 5-7--hhat
37.4
37.4
38.3
38.7
37.4
40.4
42.4
53.3
$3.8
$3.6
39.4
39.7
53.7
54.8
StarCader2 15Bre2--1B---- Ins.ruct-vO.1
37.1
37.1
37.9
38.3
37.1
39.4
40.5
52.7
52.8
$2.1
13B. Instruct
29.0
29.0
43.9
LodelLlama-78. 7B-Istrett
21.8
21.8
22.3
22.3
21.8
23.5
25.2
35.0
33.4
33.9
5.2 RESULTS ON CONYCODEWORLD
3 present MRR and Recall scores, respectively, for both closed-source and open-source LLMs across various feedback combinations. These results provide comprehensive view of model performance in CONYCODEWORLD
Overview of Results els, Llama-3. 1.t.-...--77--- 1-7OB--nntrrut demonstrated competitive Recall performance, surpassing both GPT-4- Turbo and GPT-4o in certain scenarios like (fe, [f fu> and (fe, ffl
While closed-source models generally outperformed most open-source mod-
Notably, this Recall gap between closed-source and open-source models narrows significantly under specific feedback settings, particularly when expert-level verbal feedback f is provided. For in- stance, in the (fe, o, f*) setting, DeepSeek-Coderts IB-strut (82.8) outperformed GPT-4o (82.3), and DeepSeek-Coder338B Cooee333-Innstrte (85.4) outperformed GPT-4-Turbo (84.7).
Another key observation that, among open-source models smaller than 30B, no clear winner emerges across all feedback combinations. This emphasizes the importance of selecting models based on the specific type of feedhack available.
5.2. FEEDBACK COMBINATIONS: DIVERSIFIED EVALUATION
We observed significant performance variation within the same model across different feedback combinations, emphasizing the necessity of CONVCODEWORLD for evaluating code generation models under diverse feedback conditions.
Specifically, we summarize the effect of providing different feedback combinations:
Impact Novice-Level Verbal Feedback on Execution Feedback Utilization Without novice- level verbal feedback (f.), some models Deeeelss-DeeSeee--Ceeeeeee Instruct, CodeeQwenl.5-7B-ee Codeellama77B-Instruut-----
DeepSeek-Cont Corceek--oodrr-677.. CodeLlama- 33B-Issuuut,, and minimal performance differences between partial (<fe, fer o)) and full (ffc, f σ)) test coverage in execution feedback. However, these models showed greater re- liance fu. especially in (fe, ft fo> compared to (fer fu). indicating that they need fo fully leverage f*. In contrast, high- negh-eerformin models GPT-4, GPT-4-Turbo, GPT-4o, and Llama-3.1- 70B deB-ddmmsttrated larger performance boost from (Se, Sey Φ) to <fes f Φ)) compared to the boost

---

Published as a conference paper at ICLR 2025
Table Recall results on CONYCODEWORLD. indicaaess that feedback that type is provided (ő). The leftmost results, with three X, represent f (Ø, Φ, Φ). corresponding to single-turn code generation without any feedback. For each column, bold and underscore indicate 1st and 2nd place performance within the same model group.
Complation Feedback
fc
f0
Jc
fc
fc
Je
Jc
fc
Execution Feedback
fx
Verbal Feedbacck
f
Closed Source Modell
GPT-4-0613
89.7
GPT-4-TUrbo-222444--09
48.0
48.0
56,7
63.8
48.0
58.6
68.
84.7
87.5
88.5
GPP-4o-2024405--33
50.8
50.8
60.5
67.6
50.8
60.8
69.6
82.3
84.9
86.2
Open-Source Models (e 30B)
45.4
45.4
56.2
64.8
45.4
$9.5
70.8
86.7
88.9
91.8
SeepSeek-Code-33BBBBn
45.5
85.4
84.6
ReflectionCoder-DS-s
41.6
41.6
45.3
44.9
41.6
51.4
57.2
81.4
81.8
84.2
Qwent Swenl.5-72B5 1-m11-577B--haa
32.9
33.2
39.9
47.5
33.2
47.5
57.9
84.4
86.1
87.2
5. Chat
32.0
32.0
41.1
45.3
32,0
44.6
54.3
75.9
61.8
57.1
Codel 34s-Inst
28.8
28.8
33,7
35.8
28.8
37,5
44.6
80.0
82.0
82.3
Open:
Models
I.amm-3.1 3.10--3.-8BB Instruct
Insttrecttt
31.4
31.8
38.4
40.0
31.7
43.2
51.8
80.9
80.2
83.7
DeepSeek Coder-VV -i---22-1 V2-Lite Instruct
38.3
38.3
43.4
46.1
38.3
47.0
51.4
76.3
75.8
76.9
DeepSeek- Cader-6.7P Ins-nutt
35.2
35.2
37.7
37.5
35.2
43.3
48.2
82.8
82.5
83.1
CofeetionCodee-DS--6) 7B CodeQwen1 5-77B-hha
37.4
37.4
39.6
40.7
37.4
44.7
50.4
79.1
79.6
78.9
39.6
StarCader2 15B 15cee2--1B----
Ins.ruct-vO.1
37.1
37.1
39.3
40.0
37.1
42.6
46.3
76.9
76.8
75.6
Insrrtttuctt
29.7
30,0
70.7
CodeL ModelJlam--7B-nnttnn 78-Instract
21.8
21.8
22.9
23.0
21.8
26.2
30.5
61.7
53.9
55.2
from (se: Ser Φ) to Jer This suggests these models can infer refinement strategies directly from raw execution feedback without heavily relying on fu-
Impact of Expert-Level Verbal Feedback on Execution Feedback Utilization Most models demonstrated performance improvements with richer execution feedback, progressing through the sequences (Se, Φ, & (Jer fe: J and (Je, Je f3) However, exceptions arise: (a) DeepSeek- Coder family and ReflectionCoder--S of execution feedback: (b) Llama-3. 1-r-33.1-B-BB1 I-St-tnst ReflectionCodder-SS-3333 D3B33B, and CodeQwenl
exhibited no performance difference with the inclusion
7B-Chat showed no significant difference between (fe, ftt) and (fe: fe, f but performance im- proved when full test coverage ff" f.>) was ensured; (c) In some weaker models Qwen1.5- 32B-Chat and StarCoder2- 1 ---nc-easinnnn the test coverage from (Se, Je, to (Je, J { resulted in negative performance impacts. Additionally, the highest performance of Qwen Dwven..5322P 15-33B--Chh was observed with (fe: Φ. f while adding execution feedback (fe or fz led to decreased performance. We hypothesize that weaker models struggle to utilize complex feed- back effectively, resulting in lower performance. We further discuss the possible reasons for these exceptions in Appendix C.4
5.2.2 MULTI-TURN FEEDBACK: WEAKER MODELS OUTPERFORMING SINGLE-TURN SOTA
Weaker LLMs with sufficient feedback outperformed the single-turn, no-feedback performance (Ø) Φ, o)) of state-of-the-arr models like GPT-4 and GPT-4-Turbo.
When expert-level verbal feedback (f²²) incorporated, most weaker models, including DeepSeek-Codee -.oSeek-Coder-." 6.-BB-Intttt and tion performance of fate-of-the-art single-turn models such GPT-4, GPT-4-Turbo, and GPT-4o. Additionally, with the inclusion of novice-level verbal feedback (fv) and either partial or full execu- tion feedback (fe or f(), DeepSeek-Coone SaaSSeek-CCor--Se Coder-33B 33B-Instruct and ReflectionCoder DD-333B matched or exceeded the single-tum performance of GPT-4 and
MRR
surpassed the single-turn code genera-
Recall Most open-source models exhibited significant improvements when novice-level ver- bal feedback with execution feedback (ffe, [ or expert-level verbal feedback (fee [e] fo)) was provided. Remarkably, providing execution feedback with full test cov- erage while omitting any verbal feedback (ffc, Φ)) enabled some models, such DeepSeek- Coder- Moder-V2-Litee "2-Lite-Insttruct
and Qwenl.5 1--ll..572B--C0 Ch-at,, to achieve or even
exceed GPT-4's single-turn performance.
8

---

Published as a conference paper at ICLR 2025
Spearman Rank Correlation: 0.99
Pearson Correation: 0.99
Spearman Rank Correlation: 0.97
Peirion Correlation:
Spearmen Rank Correlation: 0.95
Pearson Correlation: 0.97
0.55
ConvCrdeelendd MNR
ConuCrdelendh MAR
corvCadeilench MAR
0.625
0.50
0.50
0.575
0.45
0.45
0.550
0.40
0.525
0.40
0.500
0.35
0.43
0.45
0.50
0.35
0.35
0.40
0,45
3.35
0.45
0.50
0.55
0.60
ConvCodeWarld MRR
ConvCodeWarid MAR
ConvCodeWorid MAR
(a)
Se)
(b) (Se, So)
(c) (5)
Spearman Rank Correlation: 0.95
Spearman Rank Correlation: 0.96
Pearson Correiation: 0.97
Pearson Correationn 0.97
0.675
004-4-0633
ContyColeBendh (4BR
0.523
drarch MRR
0.623
gr--teutbo-222000000
0.600
Motelllaaaattttaooooooo Usia 3.
3.575
0.575
dreporeee wite--ideeeekooottee The Th-instruct
deeporel sDDeeDDeee Coder
2.530
devepovvio perpovee--iieeeee itteruuurnuunuoeeeeeee...
Semuel iN-eetecionncddeeee 05-330
0.525
Semet LwWeeleciaan
3.500
0.500
OwnnOwee1 5-328 Chat
Owen/wee11 5-722 chat
2,475
0.475
biccdeetancoddi 15 -02
OwanCadsOwen1 5- -hat
0.45
0.50
0.55
0.63
coddlonnrcdditan 340 intruh
WonvCodeWorld
ConvCodeWordd
coodibuncodttle 13b insruttf
(d) 12 (fe, fo)
(c) Q
(fe, f..
f.>
Figure 2: Correlation between MRR on CONYCODEBENCH (ref. CodeLlama- - and MRR on CONYCODEWORLD with different feedback combinations Q.
5.2.3 GENERALIZATION: UNSEEN FEEDBACK COMBINATION
ReflectionCoder-DS family were initialized from code a specific scenario of (fe, f f(). As a result, CeflectionCoder-DS---7..7. 7B outperformed SeepSeek-Coder-ee Baater-.7B-Instruee Instruct on [felff [ffellf(l fe). However, with unseen feedback like expert- level verbal feedback (f,") ), the performance gap narrows significantly, with minimal MRR difference and DeepSeek- Baeek--Coeelllle Cnderr-nstrucct generally outperforming in Recall. This tendency is more pronounced in ReflectionCoder- Coder-DS-333B DS333B; except for (fe, [fe] fu), SrfeectionCoder-DS--333 consistently per- formed at or below the level of DeepSeek-Codes 33B-IIntruutttt across all feedback combinations in both MRR and Recall. This indicates that training on a specific feedback combination can reduce the performance on the other combinations.
and trained to refine
5.2.4 TRADE-OFF: MULTI-TURN MRR AND RECALL
We observed that an LLM requiring fewer turns to solve problem (high MRR) may not excel at solving as many problems as possible (high Recall), and vice versa: (a) Closed-Source Mod- els: GPT-4o achieved the highest MRR, while GPT-4 had the best Recall 3 (b) Open-Source Models > 30B: Llama-3.1-70B led in both MRR and Recall.
CrepSeek-Coder-3B--nne
and
followed MRR. However, with J or f feedback, Qwenl.5-72B. Chat generally outperformed them in Recall, despite having a lower MRR: (c) Open-Source Models < 30B: MRR and Recall tendencies were similar without verbal feedback. With ver- bal feedback, Code@weel.5--7BCChr excelled in MRR, while SeepSeek-Codee V2-Lite-Instruuc (ffe, [fal f.)), and DeepSeck-Codee Coder-6.7 II-Inttruct (ffz, [o] f >) led in Recall.
ReflectionnCoder-DS.
This quantitatively confirms what some accounts observed x com

---

Published as a conference paper at ICLR 2025
Table 6: MRR and Recall results on CONYCODEBENCH using logs of CodeLlama-7B-Insteee in CONyCODEWORd X indicates that no feedback of that type is provided (Ø). For each column, bold and underscore indicate 1st and 2nd place performance within the same model group.
MRR
Recall
Compilation Feedback
fc
fr
fo
fo
Exccution Foedback Verhal Feedhack
fe
fe
f.
fe
Fv
f
Closed-Souree Models
GPT-4-0613
53.0
55.8
63.1
62.7
4
59.5
65.7
9
82.3
83.1
GPT-4-Turbo2222-00H
55.7
58.3
65.4
64.0
65.3
61.8
68.2
86.8
81.4
54.2
GPT-4p-2024 200--022-005
57.4
59,9
66.4
65.7
66.8
62.1
68.1
86.2
81.9
84.7
Spunt Models C 33BB
CoypSeck-Code-33BB Instrect
48.2
50.6
60.1
58.8
59.8
51.9
58.0
83.2
78.2
79.7
RellectionCuder DS-33B
47.9
49.9
59.5
59.1
59.6
51.2
56.2
82.2
77.8
79.6
5. Chat
42.6
45.7
54.7
54.1
55.2
47.8
55.7
80.3
76.8
78.7
Qwenl 5-328-Chat
41.1
43.2
52.2
48.7
48.5
45.7
51.4
76.2
67.2
66.8
CodeLlama- J-B-st----
37.6
50.2
78.3
72.4
Open-Source Models (<c 30B)
IJama 3.B Instruct
42.6
45.3
54.7
54.0
$4.9
47.9
SA.6
80.9
75.9
78.0
PeepSee--Coder--ee Instruct
46.3
48.4
58.2
56.0
57.1
51.1
55.6
82.0
74.7
77.9
DespSeck-Codar-4 7B-Instruct
43.0
45.4
56.5
53.6
56.0
46.8
52.9
81.3
77.5
78.7
RellectionCodee--S-
CodeQwenl. 7B Chat
45.8
47.4
56.3
55.6
56.3
49.1
53.2
78.0
74.1
76.3
StarCader2: Ins.
78.0
72.2
CodeLlama-- 3BB-sntt
34.8
36,9
47.2
46.9
47.2
37.8
43.2
73.1
68.9
68.9
5.3 RESULTS ON CONYCODEBENCH
While CONYCODEWORLD provides valuable insights into interactive code generation ious feedback combinations, CONYCODEBENCH offers a faster, cheaper, and more reproducible alternative. As discussed in $ chose Baeelllmm-7---stteeee cluded scenarios without verbal feedback, as they do not require LLM intervention. Additionally, achieved a 100% compilation success
the reference model, and
(fe, Φ, fu) scenario was omitted as CodeLlama-7B-Insree rate in the initial generation, eliminating the need for novice-level verbal feedback on compilation.
CONyCODEBENCH as a Reliable Proxy We conducted a comparative analysis of CON- VCODEBENCH and CONyCODEWORL.. the MRR (Figure and Recall (Appendix 1 results across target models and feedback combina- tions Spearman's 's rank correlations ranged from 0.82-0.99, indicating that CONNCODEBENCH is reliable, efficient, and cost-effective proxy for CONYCODEWORLDD
validate CONyCODEBENCH as proxy, comparing
Additionally, T the results on CONYCODEBENCHH showing that MRR ranking trends closely aligned with CONvCODEWoRd WORD (Table 4), with minor deviations. While absolute recall and MRR scores are slightly lower compared to CONyCODEWORLD. the rankings amongst mod- els remained roughly consistent between BONNCODEBENC and CONvCODE WORLD. Based on approximately consistent rankings across CONVCODEWORLA and CONyCODEBENCH we rec- ommend code LLMs use CONyCODEBENCH solid alternative to compare against other baselines.
CONCLUSION
This paper recognizes the need for benchmarks with diverse type of interactions conversational code generation. To address this gap, we introduced CONyCODEWORLL novel and reproducible environment designed to assess LLM code generation abilities across nine varied feedback scenarios. Additionally, for scenarios where API call costs prohibitive, we offer CONVCODEB zero-call benchmark from pre- rre-generated feedback logs, providing a highly correlated evaluation of the conversational code generation capabilities of LLMs with MoNyCODEWond contributes to more thorough evaluation of diverse multi-tumn evaluation objectives, and highlights gap invite for future models in the new design space.
Our work
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Divyansh Agarwal, Alexander Fabbri, Ben Risher, Philippe Laban, Shafiq Joty, and Chien-Sheng Wu. Prompt leakage effect and mitigation strategies for multi-turn LLM applications. In Franck Dernnncuur, Daniel Prootiuc-Pietro,, and Anastasia Shimorina (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. Miami, Florida, US, November 2024. Association for Computational Linguisties. doi: 10.18653/w1/2024.cmm weee:4eeenll-iiiuueee industry.94. URL httpps//aalantbooooo.oo///////// em. cmn.p-innduste 947
Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Do- han, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoe V. Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732, 2021. URL pattss:/aaxxar org/
abs / 22002101... 07732
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chenggiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang. Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 200,:2999..6 2023.
Kent Beck. Test driven development: By example. Addison- Addison-Wesley Professional, 2022.
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. net htpp:/dfidddas www.dkktttttmmmue
URL /tttss://peenrviiie
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder. Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavar- ian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plap- pert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Pe- ter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. CORR, abs/2107.03374 2021. URL https: org/ wtap////1..00..
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations, 2024. URL
attps://opperrewe
net/ forum Poorummiddiae wwwdddiddkuippppppe
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang. Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. I.. Jin, Ruyi Chen, Shanghao La, Shangyan Zhou, Shanhuang Chen, Shengfeng
11

---

Published as a conference paper at ICLR 2025
Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang. W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. 7. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang. Zhewen Hao, Zhicheng Ma. Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li. Ziwei Xie, Ziyang Song. Zizheng Pan, Zhen Huang. Zhipeng Xu,, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforce- ment learning. 2025.
htthtttss/////rll org/ abs/ 2501 12948
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint urXiv:2301.0023: 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprini arXiv:2407.22 2 1783, 2024.
Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you' ve signed up for: Compromising real-world Ilm Ilm-integrated applications with indi- rect prompt injection. In Proceedings of the 16th ACM Workshop Artificial Intelligence and Security, pp. 79-90, 2023.
Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y.K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. CeepSeek-Coder When the large language model meets programming the rise of code intelligence. CoRR, abs/2401.14196, 2024. BtLLhttps:/ //axxiv orw.oogg//aoooo abs www.ab//11111 /20024411..00 14196
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. In Joaquin Vanschoren and Sii-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Bench- marks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URI. http:////baasstttt--eenmmmeerrs hash/
Omar Khattab, Amay Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshay Santhanam, Sri Vard- hamanan A, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. DSPy Compiling declarative language model calls into state-of-the-ar pipelines. In The Twelfth International Conference on Learning Representattonss 2024.
net wtteee/ooooummnnsssd0o0111
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbi, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Push- meet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. Science, 37866624) 2022. doi: 10. 10.11266science.abbl 158. URL https: www. www.iiicc..........o.....o.. 1ttp:////o..111///eeeeeeeee wwtp///ccccccc.eeeeeeeeeeeeeee.s abq1158
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024. doi: 1621/al a 00638. URL
https:// antho wgw.oogg...2...cac/// 2ttpr///2224.accl111
12

---

Published as a conference paper at ICLR 2025
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self- feedback. Advances in Neural Information Processing Systems, 36, 2024.
Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar- Lezama. self-repair silver bullet for code generation? In The Tweifth International Confer- ence on Learning Representations, 2024. URL https: https://ppenevvew..neo net http://wwwwettooouuomm.m y0GJXRungR
OpenAl. Gpt-4 technical report, 2023.
OpenAL. Openai api, May 2024. URL
Jiaxin Pei, Aparna Ananthasabrammnnian. Xingyao Wang, Naitian Zhou, Apostolos Dedeloudis, Jackson Sargent, and David Jurgens. POTATO: The portable text annotation tool. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 327-337, Abu Dhabi, UAE, December 2022. Association for Computational Linguistics, doi: 10. 2022. emn ww demos...... 33
URL /tt/ttts::/acclttthoooo org/
Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models. In NeurlPS ML Safety Workshop, 2022.
g giaRo_ 7%mug.
Houxing Ren, Mingjie Zhan, Zhongyuan Wu, Aojun Zhou, Junting Pan, and Hongsheng Li. Re- flectionCoder: Learning from reflection sequence for enhanced one-off code generation. arXiv preprint arXiv:2405. 17057, 2024.
Baptiste Roziere, Jonas Gehring, Fabian Glocckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflex- ion: Language agents with verbal reinforcement learning. In Advances Neural Information Processing Systems, 2023.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
US Bureau of Labor Statistics. Table b-3, average hourly and weekly earnings of all employees on private nonfarm payrolls by industry sector, seasonally adjusted., 2024. URL (https: /tetps::/www bls. gov news. lasse eempst/ppsss..//e it
Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. MINT: Evaluating L.MSs in multi-turn interaction with tools and language feedback. The Twelfth International Conference on Learning Representations, 2024. URt.httss: httlhtttps//oppereeeet wwwpeeeeee..eeeeeee
net/ pet/foose wtw.fooruuuuuoooe 3p33wMMM111
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
John Yang, Akshara Prabhakar, Karthik R Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. In Thirty-sevnnth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. https:
dlopenreview.neee foatt/ffoumm20
i
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 7-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. URL https: // /ariiiiiiv.. org. abs/ 2406. 12045
13

---

Published as a conference paper at ICLR 2025
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma. et al. DeepSeek- Coder- V2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406. 1etp:224...199990 11931, 2024.
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. BigCodeBench: Benchmark- ing code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877,0000 2024.
14

---

Published as a conference paper at ICLR 2025
A DEEKSEEK-RI-DEE RESULTS ON CONVCODEWORLD
Table 7: MRR results on CONYCODEWORLD. x indicates that no feedback of that type is provided (6). The leftmost results, with three X, represent ? {Ø) Φ Φ). corresponding single-turn code generation without any feedback. For each column, bold and underscore indicate 1st and 2nd place performance within the same model group. Maximum token length is set 8K throughout the experiments, except for the R RI-Distlll models, which are set to 16K.
Compilation Feedback
fc
fe
Jc
fr
fc
fc
Exccution Foodback
30
Verbal Feedback
X
fu
8
1
1
Closed-Souree Models
GPT-4 0613-40063
52.1
56.4
GPT-4-TTrho--2244-4403
48.0
48.0
51.8
54.8
48.0
$2.6
56.4
62.4
64.3
64.5
GPT-4o-2024-0--13
50.8
50.8
55.0
$7.9
55.1
58.6
63.3
64.7
65.3
Open-Source
Models (3r 30B)
DeepSeck-RI-Dee DerSSct-RR-Diiill--laa----- (16K)
46.
46.2
51.2
55.2
46.2
$1.3
55.3
58.0
59.5
59.7
Llama-3.3-70B--ase
47.7
52.6
56.0
57.0
61.6
64.1
DeepSeek-RI
Rr-Setek-R--issille Distill -isstill-wenn322222 Qwen-32B (16K)
45.9
45.9
51.2
54.3
45.9
51.7
55.8
60.3
61.4
62.4
Quen2.5-328
45.8
45.8
47.9
49.5
49.8
53.4
61.6
62.7
63.8
Llama-3.1 1omm--3.-770-Immaee
45.4
45.4
49.9
53.4
45.4
50.8
55.2
60.7
62.6
63.3
DeepSeek Coder Instruct
41.6
41.6
43.4
43.6
41.6
45.5
48.0
58.6
58.5
58.8
ReflettimnCndgr-D5--333
41.6
41.6
42.9
42.9
41.6
45.6
48.1
57.7
58.2
58.91
32.9
33.0
35.8
38.3
33.0
38.6
41.4
50.6
52.0
52.7
Qwenl.5-33B-CC5
32.0
39.7
CodeLLama 34B Instruct
28.8
28.8
31.0
31.9
28.8
32.5
35.1
48.7
49.2
49.8
Open-Souree Models (<c 30B)
Llama-3.1 3.ama-3.1-88--nee --8t-trteet
31.4
31.5
34.0
34.6
36.1
49.4
$1.3
DoppScl-Codee-2-- Liedde-V--Lii--nsteee
38.3
38.3
40.5
41.7
38.3
42.0
43.8
52.7
52.9
53.3
DeepSeek- Coder-6. 6.7B-678888 18--nsttcct
35.2
36.2
ReflectionClsder DS-6.7B
37.4
37.4
38.3
38.7
37.4
40.4
42.4
53.1
53.8
53.6
CodeQwen 5--7BBhhat
39.7
42.0
43.7
53.7
53.5
SturCuler2 15B Instruct-vO.1
37.1
37.1
37.9
38.3
37.1
39.4
40.5
52.7
52.8
52.1
Codel Instructttt
28.4
28.4
29.0
29.0
28.4
31.2
33.0
43.9
44.3
44.8
CodeLlama-7B-Innse
21.8
22.3
25.2
35.0
33.4
33,9
CONNCODEWWEEE
Table 8: Recall results CONYCODEWORLD X indicates that feedback of that type is provided (ő). The leftmost results, with three X, represent S (Φ, Φ Φ), corresponding to single-turn code generation without any feedback. For each column, bold and underscore indicate 1st and 2nd place performance within the same model group. Maximum token length is set to 8K throughout the experiments, except for the RI RD-Dill models, which set 16K.
Compilation Feedback Execution Feetback
X
fc
fc
fe
fc
fc
x
fx
fr
fe
Feedback
fm
fo
Closed Source Models
GPT-4-0613
46.0
46.0
60.3
70.5
46.0
61.9
72.5
89.7
91.1
92.5
GPT-4-Turho-2224444409 GPT-4o-2024-05-13
48.0
56.7
63.8
48.0
58.6
68.
87.5
88.5
50.8
50.8
60.5
67.6
50.8
60.8
69.6
82.3
84.9
86.2
Open-SSource Mödels (3s 30B)
DeepSeek RI Distill Lama 70B (16K)
46.1
46.2
61.7
72.7
46.2
60.2
73.8
82.0
86.8
86.1
Llama-3.33 Innsseetttt
47.6
47.7
59.0
67.7
47.7
61.5
72.2
84.6
87.6
88.9
SoysSetkRR--istil--eeeeee (16K)
45.9
68.1
45.9
61.2
74.0
Qwen2.5-325 5-32B
45.8
43.9
50.4
53.9
46.0
$4.88
62.6
84.7
85.5
87.5
708
41.6
41.6
45.5
46.1
41.6
50.4
56.6
85.4
84.6
85.6
RedlecionnCder-DSS33
41.6
41.6
45.3
44.9
41.6
51.4
$1.4
81.8
84.2
Oweal.5-72B-hhas
32.9
33.2
39.9
47.55
33.2
47.5
57.9
84.4
86.1
87.2
Qwent 5-32B-hhat
32.0
32.0
41.1
45.3
32.0
44.6
54.3
75.9
61.8
$7.1
CodeLlama-344- 3-B-rsstuut
28.8
82.3
Open-SSurrce
Models (<: 30B
Llama-3, Llama-3.1-88-Imme 1-8B-Instrrct
31.4
31.8
38.4
40.0
31.7
43.2
51.8
80.9
80,2
83.7
DeepSeek- DeepSeek-Coder-Ve Voder-V2-Lite Lnte-rrstt
38.3
38.3
43,4
46.1
38.3
47.00
51.4
76.3
75.8
76.9
DeepSeek Coder 6.7B. Insiract
35.2
35.2
37.7
37.5
35.2
43.3
48.2
82.8
82.5
83.1
ReflectionCoder-DD
CudeQwenl 5. 5B78 Chat
39.3
39.6
40.1
41.1
39,5
45.8
49.5
74.4
74.7
77.4
StarCader2- Instruct--O..
37.1
CodeLlama- I3B-rutt
28.4
28.4
29.7
30.0
28.4
35.1
41.1
69.0
70.7
71.6
CodeLlama- Bnntstrtt
21.8
21.8
22.9
23.0
21.8
26.2
30.5
61.7
53.9
55.2
15

---

Published as a conference paper at ICLR 2025
Table 9: Hyperparameter tuning results of DeppSeck-I--Distil----e----- BigCodeBench- Hard-Instruct.
Temperature
Max Token Length
Pass@1
Reported
43.9
0.0
8K
43.7
Reproduced
0.2
16K
45,9
1.0
8K
44.6
1.0
16K
45.1
We present the WOONCODEWWRLLY results for the DeepSeck-R1 Ri--sstil (DeepSeck-A) Al et al. 2025)
models. These models, which are trained to handle more complex reasoning, required the following hyperparameter adjustments for inference:
Increased Token Length: The maximum token length was increased from 8K to 16K (see Table support longer reasoning steps.
The temperature was changed from 0.0 0.2. The 0.0 setting resulted in degeneration, causing repetitive sentences in reasoning. We also experimented with temperature of 1.0, ol models only support this value but 0.2 provided the best performance.
Temperature Adjustment:
Tables 7 and 8 extend the results presented in Tables and [5] including two DeepSeek-RR Miibeek-R1-Ditttee -iiistllll models (Llama-70B and Qwen-32B) and their base models (.lama-3.3-70B-Insttn and Qwen2.5- 32B). We summarize the key observations on the impact of RI :I-Distillation:
Lack of Significant Improvement: R1-Distilled models do not demonstrate a significant improvement over other models.
Reduced Expert Feedback Utilization: R1-Distilled models face challenges in effectively utilizing expert feedback over their base models.
DeepStekk-R--Distil--laaaaa
vs. Llama-3.3-70B. 3.n----rruttt
MRRR: R1-Distillation results a decrease in MRR performance.
-
Recall: B1-Distillatiin generally improves the utilization of execution and novice feedback but hurts expert feedback.
SeapSeek-RI--Distil----eeee
vs. Qwen2.5-32B:
MRR: RI-Distillation improves the utilization of execution and novice feedback but slightly hurts expert feedback.
Recall: RI-Distillation improves feedback utilization across all types of feedback in most feedback combinations.
B DISTINCTION OF CONYCODEWORLD
We elaborate distinctive implications from existing works such as InterCode Yang et al. (2023) and MINT Wang et al. (2024)
Comparative analyses of partial to full test coverage in execution feedback enables to eval- uate both:
Test generalization: A model's ability to produce code that passes full tests even when only partial tests are provided.
Test utilization: model's capability to leverage given test results for code refine- ment.
InterCode-which uses full test only evllatvsl test utilization only, and MINT-which uses partial test only-provides an entangled evaluation of test generalization and test
ht httpp::/ccommutty.pppnss
model 1ooellls--i
16

---

Published as a conference paper at ICLR 2025
utilization. In contrast, CONYCODEWORLD. by providing both partial and full test, ables isolated evaluation each test generalization and test utilization as we illus- trate below. For instance, in Table | SrppSeek-Code--... IB-Instruct shows modest test generalization ((fes d, Φ) - (cc fe. Φ) 35.2 - 37.7), but shows limited test utilization (Se, fe; o (Jes S :): 37.7 37.5). Meanwhile, Llama-3.1-885 I-St-uuttttut exhibits pabilities in both test generalization (e Φ) )) - <fc: o): 31.8 - 38.4) and test uti- lization (fer fe: o) (Ser f 0): 38.4 40.0).
CONyCODEWORLD simulates an "engaged" user, offering verbalized explanations of test results, as illustrated in Figure 17 In contrast, InterCode lacks verbal feedback, and MINT provides only generic feedback for fu--"Yuw answer is wrong. To evaluate how ver- balized explanations enhance models' test utilization capabilities, an exemplar scenario in CONYCODEWORD is when both full execution feedback and novice feedback are avail- able. In Table 5
Without the inclusion of novice feedback (fe: f x)): Llama-3, 3..-.B..1BBBB. 1-.---8-Bsntiie Instruccss test utilization capabilities (40.0) are weaker compared to CodeQwenl 1areen1..-B--hhh (41.1). With the inclusion of novice feedback (Se: 1 f.)): significantly improves Llama- 3.1-8B- performance, surpassing CodeQwent waadeeeeenl.-777 -5-7B-Chat (51.8 vs. 49.5).
Covering comprehensive combinations of feedback types, CONVCODEWORLJ analyzes previously underexplored cases, such as:
Full execution feedback vs. partial execution and novice feedback
Partial execution and expert feedback vs. full execution and expert feedback
Full execution and novice feedback vs. expert feedback
Cost-effective static benchmark (CONsCODEBENCH): CONYCODEBENCH correlates strongly with online evaluation while reducing costs. Neither MINT nor InterCode pro- vide such a static benchmark.
C VERBAL FEEDBACK
C.I DISCUSSION ON EMPLOYING LLMS FOR VERBAL FEEDBACK GENERATION
key challenge in creating CONyCODEWORLD is generating verhal feedback. Human annotation both impractical and inconsistent (3..1.), which led to employ GPT-4o for this task. While GPT-4o may not fully replicate the nuances of human feedback, ensures reproducibility and af- fordability, both critical for maintaining consistency across benchmark evaluations. As demonstrated by direct comparisons between LLM -LM-generated and human feedback in prior studies (Wang et al., 2024). we find this method sufficiently effective for our benchmarking purposes.
C.2 COST-EFFICIENCY CONYCODEWORLD COMPARED HUMAN ANNOTATION
In the worst-case scenario, CodeLlama-7B: Instrutt which requested the most verbal feedback due its low performance, incurred a total cost of $215 (26.4M input tokens and 5.5M output tokens) for 15,905 turns using EPTT-4o-2024-05 03-1-3 pricing (S5/IM input tokens and $15/1M output tokens). By comparison, assuming human annotation takes 96 seconds per turn (Wang al. 2024) and the average U.S. private non-farmer worker's hourly wage is $35.04 according to US Bureau of Labor Statistics 2024), the human annotation cost would be approximately $14,792.
C.3 HUMAN EVALUATION OF GENERATED VERBAL FEEDBACK
Table Human evaluation of simulated expert-level user feedback by GPT-4o and real user feed-
back by ShareGPT.
Expert Feedback by Is Helpíul
HlmmanFFxpert-L.ik
StareGPT
35%
DONYCODEWORLD
55%
25%
We conducted human evaluation to validate the realism of simulated expert-level user feedback, not- ing that in-context examples might lead to unrealistic responses. Specifically, human evaluators
17

---

Published as a conference paper at ICLR 2025
rated randomly assigned feedback samples from either real user feedback from ShareGPT logs expert feedback generated by CONYCODEWORD using GPT-4o (see Figure 19 for the annotation platform). As shown in Table T0 generated feedback was found to be comparable to authentic logs in terms of expert-human-lieene and was rated higher for helpfulness, consistent with MINT"s findings.
C.4 POSSIBLE REASONS FOR THE OBSERVED "STRUGGLE TO UTILIZE FEEDBACK"
From Section 5.2.1 we further discuss two possible reasons for models when they struggle to utilize complex feedback:
Limited Model Size: Smaller models, such as ReflectionCoder-D Do---ood-r----..--- may lack the capacity to process and integrate complex information effectively, which could limit per- formance improvements even when execution feedback is included (35.2 37.7). In trast, their bigger versions like ReflectionCoder-D Coder-DS-3B demonstrated performance gains with execution feedback (41.6 45.3). Mixed feedback types may distract small mod- els further. When comparing Expert feedback only vs. Expert feedback execution feed- back. For Qwen 1.5-Ch,,, the 72B model's performance improved with execution feedback, while the 32B model's 's performance deteriorated, which suggests that smaller models might become distracted when faced with multiple feedback signals simultaneously (Liu et al.) 2024 However, this distraction may be mitigated with well-designed training data, even smaller models like 1ama-3.1-8P -B-Instuct show improvements when provided with more execution feedback.
Limited Generalization Training: ReflectionCoder models were trained on specific feedback combination, (fe, Je fe), limiting their adaptability to other feedback types (Section 5.2.3p. For example, with expert feedback, ReflectionCoder-DS scores lower (85.4).
(81.4) than its base model
C.5 ANALYSIS OF GROUND TRUTH CODE LEAKAGE IN GENERATED EXPERT-LEVEL VERBAL FEEDBACK
Table 11: Pass results of various LLMs with expert-level verbal feedback f generated by GPT- 4o compared to direct ground truth code feedback. The total number of turns = 1. For each column, bold and underscore indicate and 2nd place performance while keeping the code gener- ation model fixed.
Feedback
Code Generation
GPT-4406611
GPT Turbo 2024 04-09
GPT-4a
w/o Feedback
46.0
48.0
50.8
+ Leverl-Level Verbal Feedback
70.0
69.0
68.5
Ground Truth Code
97.9
88.2
79.7
Table 12: Ground truth code leakage ratio (%) by incorporating different models for expert-level verbal feedback generation. The lower the better.
Mentioning
Including
1 Generation
ground
(L)
Refined Code (L)
GPT-4.0613
51.1
0.0
GPPT--TTurbo22244-1100 GPT-40-2024-05-13
31.4
0.0
2.5
0.1
The generation of expert-level verbal feedback / involves comparing the generated code with the ground truth code to provide modification suggestions. This process could raise concerns about potential code leakage. As shown in Table [I] providing the ground truth code significantly outper- forms providing f empirically confirming that f* is unlikely to be a direct copy of the ground truth code.
1 cuna_unfi tee
ed/ o01ob/maal htwpn//Shaaaapaee infiltered cleaned sp1 it insoryyjaa
18

---

Published as a conference paper at ICLR 2025
To detect leakage,
canary sequence approach, commonly used test for training data prompt leakage in LLMs (Team et al. 2024, OpenAl 2023 Greshake et al.) 2023 Perez & Ribeiro
2022 Agarwal et al. 2024). Specifically,
consider leakage if the feedback simulator includes canary sequence within the feedback. This sequence contains the term ground truth code, which is given in the prompt (see Figure [5). As shown in Table leakage rates are estimated by how often a model references the ground truth code in f For example, a leakage might be detected if the feedback contains phrases such as, "Unlike the ground truth code, the current code omits exception handling of DivideByZero... (see Figures 20 and 21 for comparisons of desirable vs. leaked cases).
Notably, GPT-4o shows the lowest leakage rate 2.5%, indicating its ability to generate f" with minimal leakage. This suggests that when f generated by GPT-4o is provided, the observed per- formance improvement not driven by exposure to the correct code.
C.6 COMPARATIVE ANALYSIS VERBAL FEEDBACK ACROSS DIFFERENT LLMs
Table 13: @ass@1 results over different model combinations of expert-level verbal feedback f gen- eration and code generation on CONvCODE WORLD where ? (fe, o, f and the total number of turns = 1. Each row represents model used to provide verbal feedback. Each column represents a model that utilizes this feedback refine code. For each column, bold and underscore indicate 1st and 2nd place performance while keeping the code generation model fixed.
1 Generation
Code Generation
GPT-4-0613
GPT-4o-20224055-3
w/o Feedback
46.0
48.0
50.8
GPT-F0615
63.
61.4
GPT-4-Turbo Tet-T-oooo222-0--402
62.9
59.9
62.5
G0T-40-2224006
67.1
65.4
64.2
In our main experiments, we utilized GPT-4o for verbal feedback generation and investigated its performance in comparison to other models. To see the effect of using other LLMs for verbal feed- back generation, we conducted single iteration of code generation using three closed-source LLMs both code generators and expert-level verbal feedback generators, examining the Pass@) I perfor- mance. Table [3 evaluates different models as potential verbal feedback simulators. The effective- ness of the feedback provided by each simulator assessed by comparing the performance across columns, showing consistent superior performance when employing GPT-4o for feedback genera- tion.
D VERBAL FEEDBACK BY OPEN-SOURCE LLMs
Table 14: Pass@1 1 results over different model combinations of expert-level verbal feedback f generation and code generation on CONNCODEWORLD where (Je: Φ, I and the total number of turns 1. For each column, bold and underscore indicate 1st and 2nd place performance while keeping the code generation model fixed.
Code Goneration
1 Generation
GPT-1o-0024-05-13
Llama-3. 1-70B-Instruct
w/o Feedback
50.8
45.4
201--4--2024-0--33"
64.2
63.1
Llama-3. 11-708-Instrus
65.8
62.1
Table 14 supports the feasibility of using 1lama-3.1-70B-Insttii as a verbal feedback simulator, replacing GPT-40-2024-05-13.
E IMPLEMENTATION DETAILS
We utilize DSPy (Katttab al. 2024 manage the interactive code generation flow for CoN-
"CODEWORLD and CONyCODEBENCH For both code and verbal feedback generation fol-
low DSPy's default prompt format, incorporating ChaingOrThought (CoT) (Wei al. 2022)
https://github.com htt.tttibb.ccomtttttooooo//////d
19

---

Published as a conference paper at ICLR 2025
Table 15: Pass @ results over different implementation for initial code generation without feedback. CONvCODEWoRd chose Direct Generation by BigCodeBench implementation, which showed the highest performance. For each column, bold and underscore indicate 1st and 2nd place performance while keeping the code generation model fixed.
Implermentation
DeepSeck Coder Insiinnstruct
GPT-40-2024-055
Feedbback (.)
Reported
15,5
51.1
Direct Generation C Tmppl."
Fredict
33.6
1.8
yopy. BeeeyChhenOOOThogaae
20.2
49.3
Compilation Feedback only
(Je: 4: 9)) 1)
Direct Generation (BigCodeBeach impl.)
35.2
50.8
DsPy. D8Py.2redict ChByyChhiiof
33.7
50.1
32.8
50.5
reasoning by D.Py.Cher www.Chaainofiio nofThought function. The exception is initial code generation, where adopt BigCodeBench's (Zhuo al. 2024) implementation | without CoT reason- ing. As shown in Table [5] we attribute this choice to the observation that, for initial code generation (without prior feedback), models tend perform better without additional reason- ing steps like CoT (DSPy. wwwinhhhhooughhiii or prompting (both in DSPy ESPy.Prees Predict and DSPy.ChainOfThought)
Hyperparameters are set as follows: We used greedy decoding (temperature 0) in all experiments, following Chen al.] (2023). The total number of turns 10, with a maximum token length of 8K for all code generation models. For models with lower token limit, we use their respective maximum length. For verbal feedback generation, we use GPT-44-2022-055-1 with token limit of 2K. Regarding the partial test coverage of execution feedback, we utilize the first three test cases, which aligns with benchmarks like HumanEval (Chen et al. 2021) and CodeContests (Li et al,
2022 providing up three public test cases.
20

---

Published as a conference paper at ICLR 2025
F
CONVCODEWORLD
(a) Ω 0.
(b) Ω fc
(c) Ω fc.
(d) 8 O.
(e) (Se, fo)
( 0
(ee f fu)
(g) (fe, Φ, f
(h) 52 (fe, fcs
GP144/ubbe-222446.00
24
GPT-4
Lima 3.1 708 Instruct
Laama 3.1
DoapSeak Coder instruct
DeepSeak Coder 7.78 Instruct
CodeectiorCodeeD000 D5-330
Reflection Code--5--.78
Qwen1.5:728.c Chat
Qwen1.5.328
CodeDwenl 5-7B-Chat
Codetlama 348 34S-tritruct
Codelibma 138 Instruct
()) 12- (se, 12
Figure 3: Iterative @ results on CONYCODEWORLD with different feedback combinations Ω
21

---

Published as a conference paper at ICLR 2025
15
No.
No. Numa
GPT-4-0613
(b) NPTT4-Turbo-202-
(c) GPT-44-2022-05--1
(d)
Llama-3.1-70B
04-09
Instruct
50
75
75
70
No. Tume
No.
14.
3umn
(e)
Llama-3. Llama-3.1-8B- 3.1-8B-
(1) DeepSeek-Cooe Coder- V2- Lite-Instruct
(g) DeepSeek-Codee
(h)
DeepSeek DeepSeek-Coder- Coder-
Instruct
33B-Instruct
6.7B-Instruct
30
75
70
Ifa 4. d)
70
Ife fer 4)
Mr. fg +)
Mc
Ifc. fa. fM)
(f. f... fv)
Ifc. f,))
(f. f))
Mr. f,)
N0. TuTu
No. 7m
(i) ReflectionCoder-DD 33B
i ReflectionCoaeer-DS.
6.7B
Figure 4: Iterative Pass@1 1 results of each LLM on CONVCODEWORLD with different feedback combinations Ω (continued Figure 5)
22

---

Published as a conference paper at ICLR 2025
40
30
39
1.cc2.5-72B-
(b) 5-322BChht
CodeQwen1.5-PP7
(d) Instruct-vO.1
StarCoder2-155
Chat
75
65
60
05
59
Ifc. 9. Φ)
(f.
(fc ()))
(fe. f.
(fe, fa.
Uf. fx)
(fe. 0.
(fc for f,)
2
33
(f. (())
(c)
CadeLJama-34B.
(f)
CodeL-lama-135 13BB
(g)
CodeLJama-7BB
Instruct
Instruct
Instruct
Figure 5: Iterative @ 1 results of each LLM CONVCODEWORLD with different feedback combinations 52 (continued from Figure
23

---

Published as a conference paper at ICLR 2025
G CONVCODEBENCH
G.1 MRR AND RECALL RESULTS
G.1.1 REFERENCE MODEL: DEEPSEEK-CONE 6.7B-INSTRUCT
Table 16: MRR and Recall results on CONYCODEBENCH using logs of DeepSeek-Coode Coder-667B Instruct in CONVCODE WORLD. X indicates that no feedback of that type is provided (Ø). For each column, bold and underscore indicate 1st and 2nd place performance within the same model group.
MRR
Recall
Compilation Foodback
fe
fc
Execution Feedback
f.
13
5
Verbal Feelback
fo
fm
fo
fu
fn
f
Closed-Souree Souree Models
GPT-4-0613
56.2
59.1
66.9
67.4
688
61.8
68.9
89.9
90.6
91.0
Tur44-Turb-20224
57.4
60.1
67.6
68.3
69.0
61.7
6R.3
89.0
89.9
90.0
GPT-40-2024-05-13
58.8
61.3
69.0
69.3
70.2
63.1
68.9
89.8
90.1
90.5
Open-SSurce ModeTs (a 3.0)
1amma-3.-708-Inssruet
57.2
59.2
67.2
67.7
68.5
62.3
67.0
89.4
89.7
90.4
DeepSeek 33B Instruct
52.4
54.0
63.4
64.4
65,3
56.2
60.7
86.8
87.8
88.6
DeflectinCCoder-SS--33
52.6
54.7
64.0
64.5
65.3
56.4
62.0
86.8
87.8
88.2
Qwenl.5-728-Char
49.1
52.0
61
61.9
62.7
54.6
61.8
87.6
88.2
88.8
Sweal_5-32B-Cha.
48.6
50.8
60.4
59.9
60.1
$4.1
59.2
86.3
84.8
84.8
Code 3-B-mma--BB-
47.2
48.8
60.6
61.1
56.4
87.4
88.2
Open Source Models (< 30B)
Llama-3.1-88 1-88-Instrrett
50.6
52.5
62.3
62.8
63.4
55.8
61.2
87.3
88.3
88.2
DeepSeck-Coder-V Coder-V2-Liie-Insttuee
54.4
63.1
63.8
56.4
61.7
86.2
RellectionCodee--D- Ddder-D-6677E 6.77 CodeQwent
48.5
50.2
61.0
61.2
61.8
$2.5
56.9
85.8
85,9
86.4
StarCoder2: 338-Instruct-vO.
49.7
51.7
62.3
62.2
62.8
52.9
58.1
86.6
85,9
86.6
CodeLlama: 13B Instruct
47.4
49.3
60.4
60.4
61.1
$1.8
56.8
86.6
86.2
87.4
Codel-lama-7B--nsttms In-Instratt
44.2
45.7
57.9
57.9
58.3
48.9
53.2
86.7
86.1
85.4
G.1.2 REFERENCE MODEL: GPT-4-0613
Table 17: MRR and Recall results on CONYCODEBENCH using logs of GPT-4-0613 in Con- VCODEWORLD. X indicates that no feedback of that type is provided (ø), For each column, bold and underscore indicate 1st and 2nd place performance within the same model group.
MRR
Recall
Compilation Feedback
fc
fo
fo
fc
f
fo
fc
Execution Feedback Verbal Foedback
fe
J
f.
f
Je
/
Closed- Spurce Models
GPT-4-Tirbo-2024444-09 GPT-4o-2024-05--3
64.1
69.9
70.9
71.6
76.7
91.6
92.8
94.2
61.6
65.0
70.6
71.5
72.3
68.6
77.2
91.9
93.0
94.3
Open-Source Mödels (3> 30B]
Llama 3.1-a-3.1-70--. Instrictt
60.9
64.2
69.9
70,9
71.5
68.8
77.7
92.2
93.5
94.6
DeepSeek-Coder-333 Coder-33R-Innsmn
58.3
61.9
68.2
69.3
69.9
66.5
75.9
91.9
93.2
94.3
ScfiettionCodee-S--333
62.4
68.8
70.0
75.9
93.3
Qwenl 1-nall5-72B-Caa.
57.5
60.4
67.3
68.3
69.1
66.0
73.9
91.5
92.5
94.2
Qwenl.5
92.7
Codel Jamalllaaa00 Jama-34R. Ins-tncttett
56.2
59.9
66.8
67.8
168.4
64.7
74.8
92.2
93,1
94.4
Open Oource Models [<c 33BB]
Mama-3.1-8B-nnsrrul
60.6
67.4
74.8
92.8
DeepSeek DeepSeek-Coder Coder-V2 V2-Lite Instruct
$8.8
62.4
68.9
69.7
70.1
66.4
75.5
91.8
92.6
93.9
DeepSeek 6.nttuu
91.2
ReflctionCoder-DD oder-DD-6.7B
57.9
61.5
68.0
69.1
69.7
65.7
75.2
91.9
93.0
94.1
CodeQwent 5-7B-Chat
59.0
62.4
68.5
69.6
70.2
76.1
91.8
92.9
94.4
StarCode22 15B- Ins
58.3
61.8
68.0
68.9
66.0
75.3
91.2
92.5
90.0
CodeL.amma 13B Instruct
56.1
59.9
66.4
67.5
68.1
64.9
74.6
91.5
92.6
94.4
CodeLlama- BB-Instuct
24

---

Published as a conference paper at ICLR 2025
H RANK CORRELATIONS BETWEEN CONYCODEBENCH AND
WONYCODEWORLD
H.1 REFERENCE MODEL: BAAILLAMA-7--NSTTRUCC
Spearman Rank Correlation: 0.91
Spearman Rank Correlation: 0.91
Spearman Correistton: 0.84
Poanson Correiation: 0.98
Pearson Correlation: 0.96
0.50
0.55
Conv ConvCadeennhh Recall
ConvCadedench Recall
ConvCadeBench Rocall
0.54
0.55
0.60
0.82
0.50
0.50
0.78
0.45
0.75
0.43
0.34
0.35
0.40
0:45
0.30
0.25
0.50
0.40
0.45
0.30
5.53
0.00 0.60
5./0
0.70
0.75
0.50
0.55
0.00
ConvCodeWerid Recall
ConvCodeWorid Recall
ConvCodeWorld Recall
(a) . Se)
(b) Ω (fe, fo)
(c) 12 (s
Spearman Correlation:
0.91
Pearson Correlation: 0.90
Carreaation 0.91
ConvCadellnnhh Recall
ConvCodellncch Recall
0.78
0.800
0.76
0.775
0.74
0.750
0.725
www.ceeioi..fcccrrrrr.o Co
0.70
0.700
0.50
0.675
OwenCCadeQeen) 5-78--his
0.65
0.70
roseteeaaiZdde ith-nrrrttt
ConvCodeWorid
ConvCodeWorld
cosettema Lem.- -eberrrttt
Ω
Je.
(e) Ω
(e.
Figure 6: Correlation between Recall on CONNCODEBENCI (ref. CodeLlama- Boeeellma-7B-IInse 7B-Instruct) and Re- call CONVCODEWORLD with different feedback combinations Q.
25

---

Published as a conference paper at ICLR 2025
H.1.1 REFERENCE MODEL:: DEEPSRRK-CODRER 6.7---66--IIN
Spearman Correlation:
0.96
Pearson Correlation: 0.97
Corelatiinn 0.94
Correlation: 0.92
0.62
0.38
ConyCodeBench MRR
CorvCodsBench NRR
0.38
66
ConvCodeBenchh MAR
0.66
0.64
0.51
0.50
0.82
0.48
0.53
0.50
0.46
0.44
0.45
0.58
0.25
0.50
0.35
0.40
0.50 0.55
0.40
0.55
0.50
0.55
0.00
ConvCodeWorld
ConvCodeWorld
ConvCodeWord
(a) Ω fer
(b) Ω (fe, fo)
(c) Ω (fe,
Spearman Correlation:
Spearman Rank Correlation: 0.94
Correiation: 0.94
Pearson Correistion: 0.95
0.68
01--1-6133
ConvCadleeend ConvCadelench MAR
ConCGodarenhh MRR
0.58
gpt-4s 2014-222200
0.66
mate- Hamma damaa 121--00 instruct
mate- Namaa dama 3-1 trst
0.64
deopoook axDoossuck bockkssckkeee Coderi/2 V2 Lito rust
deopscek aidespoook codor instrctt
Sencel www.feetionCCierr 05 338
0.62
0.52
Senoel (M.Reflect Codeer 05-6
QweeyOwent 5-720-4400 6-728-Chat
0.60
0.50
OwnnyOsent wwn
CaenrCCoeQQeerl 5-70--hhet
0.58
0.35
2.40
0.45
0.30
0.35
0.60
0.55
0.35
0.40
0.45
0.30
0.55
0.00
0.05
ConvCodeWorla MRR
ConvCodeWorid MRR
cocallu doattammiCodstamm....mm 7birstruds
(d) Ω2 = fe, fe.
(c) f2 = Se. 1C)
Figure 7: Correlation between MRR CONVCODEBENCH (ref. Boteeee--Codeer--------e---e and MRR on CONVCODEWORLD with different feedback combinations Ω.
26

---

Published as a conference paper at ICLR 2025
Spearman Rank Correlation; 0.93
Spearman Rank Correlation 0.92
Spearman Rank Correlation: 0.80
Pearson Correlation: 0.95
Pearson Correlation: 0.96
Pearson Correlstion: 0.66
0.50
ConvCodelemt Recall
0.50
0.56
ConvCadeBenchhhe Recall
0.59
0.38
CorvCodedench
0.56
0.62
0.00
0.57
0.52
0.55
0.70
0.75
0.50
0.85
030
ConvCodeWorid Recall
ComvCodeWorld Recall
ConvCodeWord Recall
(a) Se.
(b) (Se, 1
(c) (.
Correlation;
Rank Comrelaioon 0.34
Pearson Correlation: 0,79
Pearson Correlation: 0.90
0.91
ConvCoteleenet ConvCoteBencch Forall
0.50
ConnCodeBench Recall
0.69
0.59
meta- Meet Usma
202--2244-0 00-00-5-33
0.58
divepoaeio Deepp Codeeeee--oooe...... on
deeporatio wwdeeppeee.scooocm codee 311- Ita-itrruut
0.57
Semeet Lo- coleccoonnonen 05-330
Seroet Ltf.eel CoCodin 25-8
OwanOwen11 5. Chat
0.56
OwarOwent 3-328 Chat
OwanCaddOWen. 5-78 Chat
bigcodestarccddee 15 insrrstt-O
Codelomaaiodetbrre 34b instuctiit
5.6
0.9
codetlemaic-deeeee bama- insractift
CanvCodeWerid Recall
ConvCadeWord
Recall
(d) Ω (fe, fe>
(e) Ω
fe
Figure 8: Correlation between Recall on CONYCODEBENCH (ref. DeepSeck-Coder-6.755 and Recall CONVCODEWORLD with different feedback combinations Q.
27

---

Published as a conference paper at ICLR 2025
H.1.2 REFERENCE MODEL: GPT-4-0613
Spearman Correlation:
0.97
Pearson Correlation: 0.97
Comelatiann 0.95
Correlation: 0:93
0.65
0.51
ConvCoteBench MAR
ConuCedsBench MRR
0.64
CorvCodeBench MAR
0.50
0.89
0.50
0.38
0.57
0.67
0.60
0.59
0.25
0.55
0.40
0.50
0.55
0.40
0.5
0.50
0.55 0.00
ConvCodeWorld
ConvCodeWorld
ConvCodeWord
(a) Ω fer
(b) Ω (fe, fe)
(c) 0 0f
Spearman Correlation:
Spearman Rank Correlation: 0.96
Correiation:
Pearson Correistion: 0.95
0.71
ConvCadedench MRR
ConGGddeaench MRR
Gr-- 2324-03-2--000
0.71
cam-4
Uama 3.2-700..000-.-- insrust
Lama 3.1 48 Instruct
0.30
DespSook Codler/2 Lita Instrutt
0.68
DospSook Codle 33t instrct
DeepSeek Coderr 78 intruct
DS 338
RefeecianCodeeD544
0.58
Owen1 5-220-Caat
0.67
Owen1 5-18--ttt
Codetwen2 5-7B-hra
StarCoder2-113-- -...
0.35 2.40
0.45
0.30
0.35 0.60
0.55
0.35 0.40
0.45
0.50
0.55 0.00 0.05
Codstiama 3iB-intt
ConvCodeWorla MRR
ConvCodeWorld MRR
Codettama InstruccCade /B-instrud
(d) Ω2 = (se, fer
(c) f2 = (fe,
Figure 9: Correlation between MRR CONYCODEBENCH (ref. GPT-4-0133) and MRR CON- WORDD with different feedback combinations Ω.
28

---

Published as a conference paper at ICLR 2025
Speamann Correlation:
Speaman Rank Correlation;
Spearman Correlation,
0.68
Pearson Correlation. 0,09
0.922
ConvCadedeneh Pocall
0.58
CanvCodetlench Reccal
0:77
ConvCodeBendn Rocall
0.57
0.56
0.75
0.916
0.65
0.74
0.41
ConvCodeWorld
ConvCadeWonl
ConvCodeWorld Rocall
(a) Ω So)
(b) (fe, fo)
(c) 0,
Speaman Rank Conrelation; 0.49
Spearrn Rank Corretstion: 0.49
Pearson Correittion: 0.51
Pearson Correlatonn 0.81
2.3450
ConvCoreeeechh Recail
ConvCodeDerch Recall
2.8425
3,9000
fnstruct
3.9375
DeepSeeet
3.92
3,9350
0.9325
3.926
3,2300
ConvCodeWorid
ConvCodeWorls Recall
CodeLiama InstruaiC
Ω2 (Je, J
(e) Ω
(8.
Figure 10: Correlation between Recall on CONYCODEBENCH (ref. GPT-4-0613) and Recall on CONyCODEWORLD with different feedback combinations Q.
29

---

Published as a conference paper at ICLR 2025
I IN-CONTEXT LEARNING EXAMPLES
1.1 NOVICE-LEVEL VERBAL FEEDBACK
Example Input:
2 Sort a list of integers in ascending order. The function should take a 1ist of integers and return a sorted list. Ensure that the function handles negative numbers and zeros correctly. Check if the function's output is sorted list.
3 python
4 >>> sorted = cort_func((3, -1, o, 5, -10, 21)
5 >>> sorted_list
[-10, -1, 0, 2, 3, 5]
8 You should write selftcontained code starting with:
9 "python
def sunt_func (int_ :
13 Ground Truth Code:
14 "'pyhhon
def sort_func (int_listt::
return corted list)
18
Previous Code:
ppython
def sort_fune (int_list): for range (intiitiistt)):
for j in range (1en (int_liist) 1):
if int_list < int_list + 1]:
int_listlil, lnst_ 11 int 1ist + 11,
int list[j]
26
return int list
test_lise [3, -1, 0, 5, -10, 21 print (sort_ (ttss_list))
Compilation Feedback:
Traceback (most recent call last): File "tmp.py", line
35
return int_list
36
Indentat iontrror: unindent does not match any outer indentation leve!
Reasoning:
40
Let's think step by step in order to produce the novice-level "user_ feedback'
User Feedback:
43
got this crror:
44
Traceback (most recent call last):
File "tmp.py", line return list
48
IndentationErros unindent does not match any outer indentation level
think there might be something wreng with indentation the - 'return int_list line. Also, I think there exists simpler way to sort a list in Python.
Figure 11: The in-context example of fo generation where Ω (fe, 6, f.).
30

---

Published as a conference paper at ICLR 2025
Example Input:
Sort a 1ist of integers in ascending order. The function should take
list of integers and return scrted list. Ensure that the function
handles negative numbers and zeros correctly. Check if the function! output is a sorted list.
3 ""pyhhnn
>>> sorted list sort_funcc(3, -1, 0, 5, -10, 21)
>>> sorted list
[-10, -1, 0, 2, 3, 5]
7
You should write self-contained code starting with: "python
def sort_func iist
12
Previous Code:
"python
def sort_func (int_list):
16
for i in range (len (int_list))=
for j in range (len (int_list) 1):
if int_list[i] < int_list + 1]:
[i]_ist[j], int_listl] 1] int list + 1],
int [is
20
return int list
test 1ist 13, -1, 5, -10, 21 print (sort func (test_list))
Compilation Feedback: No syntax errors
Execution Feedback:
[GST_CASE_1
Traceback (most recent call last):
File testt_ssp"" 1ine 78, in test_case_1
AssertionError: sort_func ( [3, -1, 0, 5, -10, 21) [-10, -1, 0, 2, 3, 51
Reasoning:
Let's think step by step in order to produce the novice-Level "eeer__eedback'.
37
User Feedback: got this error:
Traceback (most recent call last): File test___.",,, line 78, in test_case_
AsserticnError: sort_func ([3, -1, 0, 5, -10, 2]) != [-10, -1, 0, 2, 3, 5]
It seems
function didn't sert the list correctly. The list descending order instead of ascending order. Also, I think there exists simpler way to sort list in Python.
Figure 12: The in-context example of fo generation where Ω = fa(
31

---

Published as a conference paper at ICLR 2025
I.2 EXPERT-LEVEL VERBAL FEEDBACK
example input:
2 Sart a 1ist of integers in ascending order. The function should take a 1stt intagers and sorted list. Easure that the function. handles negative numbers correctly, Check if the Eunctien's output is sorted list.
""python
4 >>> corted_lie List fune -i, 0, S, -ia, 2,)
sorted
10, 1, 5, 2, 3,
should
celf-contained starting with:
9 "python sort_ list
Greund Truth Code:
"ppyhhon
funt_ (int_1ist)
15
return sorted(int_list:
18
Previous Code:
*** pythen
21
def sort_funce Lint
zange (int listt)
for in range (len (int_ 1):
int_list()) ins_list: 1;: int_liat(] 1J, int_list(]]
int_liatiil, int_list
return list
Lest_1ist [3, 1, 0, 5, 10, 2] 28
print (sort_ (test_list)
Reasoning:
Let's 's think step by step in order to preduce the user feer__ee. We need to cempare the "previous_code" with the crruts_eruti_code and identify specific where the - previous__ode" deviates from the expected behavior. We should focus on the secting zeros,
algorithm, handling edge cases negative performance of the code.
overal:
Uses Feedback:
1. sortting Algorithmre: The "previous_com" implements a bubble sort algerithm, but it sorts the list descending order dua the condition 'if int_list [i] int__istt 1]". The correct approach ahould use "if int_lfat > int_list 11" to sort in ascending order.
-
38
2. Fddge Case Handlingse: Whi the "prev' pod_" attempts to handle a'l integera, it incorrectly soits the 1ist in the reverse order, Basure that the sorting condition correctly comperes elements to schieve ascending order.
AaCode simplicity and Rtyss is more complex and leas readable compared to using the built-in "sorted function. simplifying the code use "sortad()" will
buba's
implementation in "previous, "oodoe"
resdability and maincaincbility..
1. f The bubble sort algerithm used in the srevious_codet has time complexity of (("2)), which be inefficient for large lists. The "sorted()) function, on the other hand, usea rimsort, which has a time complexity of oin log n) and is more subtable for sorting large
43
Test Case Varl ficationns:
"previous_eene code" includes a print statement to verify the Lon'tion's output. While this is useful for debugging, consider using assertiona teat functions to automate the verification process in a more scalable way.
Overall, the 'previons_code" functions incorrectly by sorting the 1ist descanding order. buable condition utilizing Python's puilt-in
Using the correct comparison sorting capabilities would improve the function's correctness and performance.
Figure 13: The in-context example of f* generation where Ω (fes f*).
32

---

Published as a conference paper at ICLR 2025
example input:
Sart of integers ng oreer. The fisnction take integers return sorted liat, Ensure that the Iunetion. handles negative numbers and seros correctly, Check if the functien's cutput is sorted 1ist.
""yython
4 >>> sorted_list - sort__une((3, 2, 0, 5, 20, 21)
sorted_list
[-10, -1, 9, 2, 3, 5]
You should write self contained code starting with: ""python
def cort__anneist__lsst)))
Ground Truth Coda:
"ppthhn
def lint
return sorced 16
18
Previous
"python
def sort _ort__uun (int_
for i range (int_iiiitttt]] for in renge (len (int (int_list) 1):
if 11tt (3) ins_list() 1::
25
int_liatiil, 1ist] 1. int_l 11, int_list(]]
return lnst
test_ 13, 1, 5, 10, 21 print (sort_fune (test_list)
32
Execution Feedback:
TEST CASF_1
Traceback (nost recent call last):
File _estt_pp""" line 78, in test_case_1
AnsertionError: funt_ ([3, -1, -10, 2]) != [-10,
Reasoning:
39
Let's 's think step by step in order to preduce the user_feedbacck. We need to cempare the and ident ity apecific areas where the "previous_code" devietes from the expected behavior. We should focus the serting algorithm, handling of edge cases like negative numbers and zeros, and the overal" performance the sode.
"previous_code" with the "ground_truth
Jser Feedback:
42
1. **Sorting Algorithme+: The "previous_code implements a bubble sort algerithm, but it sorts the list descending order due to the condition "if int_list(i] int_list(] 11", The correct approach should 'if int_list(j] int_list[] 11" to in ascending order.
+ddge Handlingse: While the "previous_nen" pode" attampts handle all integera, it incorrectly sorts the 1ist in the reverse order, Easure that the sorting condition correctly coapares
ascending order.
Caddd simplicity Readabilitysus
bubble sort implementation is complex and less readable eempared using the built-in "sorted functien. simplifying the code to use "sortad()" will onhance readability and minnsainbiiitty.
'previovs_coda
48
4. Per The bubble sort algorithm used in the "previous code"" has Lime complexity ( ()),), which can inefficient on the other hand, uses Timsort, which has time complexity of oin log n) and is more sut tabla for sorting large datasets.
large lists. "sorted()" function,
Tast Case Vari Prrtfications:: The "previous_cone code" includes a print statement to verify the Function's output. While this useful for debugging, consider using assertiona functions to automate the verification process in more scalable way.
overall, the "previons_code" functions incorrectly sorting the list in dessending order. Using the correct compartson the buable sort condition utilizing Python's suilt-ir. sorting cepabilities would improve the function's correctasas and performance.
Figure 14: The in-context example of f* generation where = fe
33

---

Published as a conference paper at ICLR 2025
You are given Input, previous_com coodce execut ion "previous__oos
feeedbac to simulate uaer feedback that compares the "gcound_truth_coom
execution
2 Your task ia to provide the simulated "user_ "eer__tedbscc" that highlights specific areaa where the "previous_cote" code"" devietes fzom the "groum__iuuth_oode and auggeats improvemente oz correctiens.
You SHOULD NOT leak 'ground_crurh_code in the simulated user foedback.
generata updated code.
S Do not reveal that you can access the "grohnd_truth_ only indirect information is allowed.
Figure 15: Prompt used for expert feedback generation in the feedback combination Ω (fe. Ifll
34

---

Published as a conference paper at ICLR 2025
J CASE STUDY
Input Code
import smpplib
from email mesaage import Emoil
import getpaas import socket
SERVER SERVERLADRRESS "localhest"
SERVER_PORT 25
sue PUR__812 1024
SER_RE *sntp.gmatll s.com"
10
SMTF_PORT 587
11
12
def Fank_ (client_ Insure the socket is connected if not client_sccket:
13
14
15
print return
16
17
message
client
19
try:
message cllent_socket-t except socket connt_ssssttttte bocket.recv recv (RIFFRR_ ddoddd print
deandd
21
22
23
return
24
message expty, sot_sondinnnngeenlll... if not mesgage:
email
26
27
print wtwpro__aaemtt_eeeeeeeee return
28
29
30
for sender's email, zecipient's authentication
sender's emall password
31
sender_emett input ("Enter sender's, email;,,")
32
recipient_emal input (*Fncer rect plent
plentt 's_emat 1:_"""""""
pasaword getpass.gelpas ("Bnter__ 's paassasss
34
35
Create the emall measage amail_message EmailMessage()
36
37
www.aaa.set_comeee.eeeeee amail_masean (aassaaa""*Saa ("Subject"] email [assil_mesaaeeeee ["From"] sender_ema!]
(message)
39
email_meesage ["Io"] recipient_email
41
42
Connect Lo the SMTP server and send the emall try:
43
44
SMIP with smtplib. SMIP (SMTE SERVER, SOTE_FORT) amtp: smtp.startrlan
46
amtp, login (sender_emali, gаasword)
47
48
print (etniill,entt,ucceesstlllll.
49
except Exception as print paraalee__tt_sen_mmmiil.....
Execution Feedback with Partial Test Coverage (fe)
Traceback (nost recent: call 1ast):
3
File http:///w/.nii.ooeeeee/eeeeeeoee Func(xnevargs, snnewkeywargst
httphh/h/...pptto...... /wnittest/noottt py"" Line 1335, in patcned
File _1yt_.p"" " line 125, _est_ant p_ perpppeeres_oomeee
File test bpp". "py", Line 39, tack_tune
File "/tng/utilaage la.py", 1ime 251, , in readling me 10Ecror
11
12
TEST. SUCCESSFL, SMAT5, SEND
last):
File 15
condo/ http://awoooooooo/////ooe/eeeeee return tunc(anevarge, senewkeywargs)
line 1335, in patcaed
File thsst -tstt-..""" line 81, in text aucceesa [eeil sendd 16
testtt_.pp py", File "/tng/utileepp" line 251, in readline
task_tunc
19
zaise 10E:ree
osErsor
Figure 16: Case study of GPT-4 on CONVCODEWORLD (Ω = (Ses Su); 1 = 7): Input Code and fe-
35

---

Published as a conference paper at ICLR 2025
Novice-Level Verbal Feedback (fr.
...
Traceback (nost: recent last):
File (uno(anevargs, wntewkggaargs
http:///whoooo../attttttttteteee /tw.tttst///ock......t// Line 1335, in patcaes
File -Py" -py",
test_.nnt pu. p_aerver_ccomeee
File test toy line 39, in tuck_tunc
8
File "/tmp/utila.pp" 1a.pp" 1ine 251, in readline
10E:rer
12
Traceback (nest recent call last):
File .py Hine B1, httpe//eeeeeee.ii/////eeeee//eee h/t.:///////ttttt/oooo/oooppp 16
13
File (hateeacck//iinioon
Line 1336, in
funo(snevarge, senewkeysargsl
File test test__..p" -py,, line 39, in sank_tunc
17
File //tmp/utila.pp/ 1ine 251, in readline
2315e
oEErzor
20
like
connection the handling socket, "tack_tunc"
exceptione are seing during
Refined Code by Feedback
import amtplib
from email. import Emai INesage
import getpaas
import socket
SERVER
SERVER_ADORESS ADDRESS
"localhese"
SRRVER_
BUFFER_8123 1024
SXIP_SERVER "smtp.gmail. gmail.comm
SMTP_PORT
11
task_ func [ellient_pocket]]
13
Ensure the scoker is connected
not elient_sccket: print i "ontectce_ return
15
16
17
18
Receive the message from the client try:
20
message @lient_sockkt.raa recv SUFFFR__ST2R,.dcco except socket.erzor as e:
21
22
print her.elee_tt..rrriivveeessssssss return
23
24
25
if the message is expty, da not send the email not message:
27
print ("Received, empty, pesaage, Not, sending.emall.. return
28
29
30
Ask for the sender's amail, recipiant's 's email, and sender's email password for authenticstion
31
sander_emcis = input seseessssdddssseeee s_emaill=""
reatpient_emal input "enser,,reft reat plent 's_emal 1: pasaword getpass. sespass.getpasa ("Enter, sender 's, eaal,,
33
34
35
Create the emsil email _al_lmmasagaa Fmai Massage
36
37
email_messanl wwwessage.ee__conteeeeee (message)
38
masil_mmssaaalll ["Subject"] "Moosagg__roo_olloors
email email_measage ["From*] sender_ema
40
smail_messed ["Io"] recipient_email
41
42
Connact
and send the ema!l
43
try:
44
with smtplib, (SMPP_ _ERVER, SMTP_FORT) amup: smtp-starttlall
45
amtp. (oe.diinnnneeer.... sender__email pаesword)
47
metp.eeen__esssaeeeem (emes
48
print sitmill__ett_uuccessullll
49
except Exception
50
print [FdFFilee_to_eee benn__maill__ee1
Figure 17: Case study of GPT-4-0613 CONVCODEWORLD ( (ee tum 7): f. and Refined Code.
36

---

Published as a conference paper at ICLR 2025
Input Code
import matplotlib, htpplottbb.pppl.... pyploot plt import munpy
task_ in, seed=():
seaddddm.eeddaeeee... (seed)
points [(x, y) for X, y in np.rancom, rand (n, 2)
fig, ax subploto))
ax.scatter for y points), for x, in points])
ax.set_title ot.set_titlee$scatee__ pag.rsrr__ltt_ee__addmmooooo ax.set_xlabes (*x")
11
ax.set_ylabel
("Y") (*Y")
12
13
return fig, points
Expert-Level Verbal Feedback (f**)
Rondomm Foints Coneration The "previous_nows pode"" generates randon points using a comprehension with sap.sanon..aad 21", which is valid approach. However, simplified by diractly using "rp.random, (n) for both x and y coordinates seporstely. practices.
be
makes the code readable and aligns with
2 *ellotting Process.::
"previous_cee code" correctly plots scatter plot using "ax. sxx.scrtee sowever, it extracts x and coordinates from the "paints" list using list comprehenstons, This step can avolded by directly vaing the arrays generated for coordinetes, which simplifies the code.
"previous_come code" correctly returas fiqure and 1ist of points. Blowevez, the generation of points can be st.remlined by using "zipxx, y) dizectly on the generated x and y arrays, which
Valueate;
efficient concise.
Code Simplicity and Readabilltyya: The "previcua_ooes _ode" functioral but can be dimplified for better resdability. Using diract arrey operations for and y coordinates and avoiding unnecessary 11at comprehenatons wt wl make the code cleaner and easter to
-esppoodcciiiittyy::
expected implementation correctly ses the random saed for reproductbilstyy which ia good practice.
'previous_code
Overall, while the "previous_code" is functionsl, it can impreved for simplicity and - readability by directly using the generated x and y arrays for plotting and point generation. This
moze concise maintaineble.
Refined Code by Feedback
import matplotlib, pyplot import numpy aa op
Lask_func in, seed=0): s..rsnddm.sedd/eeee x, y np.random. rand poiints = list yl)
plt.subplots() ax.acatter (6, y)
ax.set_title (Baaaaa ax.set_xlebel ("x"1 ("x"1
11
ax.set_ylabel. (*y")
13
return fig, points
Figure 18: Case study of -oobSeckk-Codr-6.788 Instructt on CONvCODEWORLD (Ω = (,
turn 1).
37

---

Published as a conference paper at ICLR 2025
Expert Feedbnos Analytts
Home
Stntitee
Help
Friited 2000
Cumerti
Cametty oppeo in
(Problee]
you script
(httpx://mw.. erg.uki
saport
saport
fron setentum eelencewveebrrieee saport Flinefox
From selencsm, semniss.eeeeii weber aver service inport Service From setent uebdr tver, common.lyy by import By
geckudriver. "ygeth/ts/gnkheriiver" Senvioeggeckoriiee
optioon. argunent
driver Firefeac((eviioomeeeviee optiomeaptions)
def play_sound(
os.systeet lapplay /perht/o//suss.rpp
while Trams
draver- "ttt""tttps////eeee wawbaaar..arrn.. cariicarrrrrr.o wwtp.u///netttonn//ee http://aawnn/t/////ee//oooloaaa www.-/hhweeeseeeeeeeeeeeoe eeg.-eaaaaae-naoaotttrroo-ee-eoe
check dates, buttinn driver. rind etement sPATi, '//ateert(=" dates booc"]")
gave
appeor
sold_out_button driver, find benneseeeeeetttbyyeeee.. 1//butteffextt (eolos outr11)
sold, buttonss
flay_soundi break
else:
arirt("All tuttons sold out. Retrrying.") Sloe-sle:ppE0)
driver-qutt(1
[Eepett Feedback)
which the text "Boid out; the gool here to check any those buttons I'do nott" have that tent, and then clickit. What you've done look see whether there t'any)? buttons sold out, you kind of have it
modifiattona
round. Can again plusss, with the (ie, our button), playlig me the sumoou ding tes about substiiuting paths, lenow what I'm doirgg
that
clicked
give
retining
espert programme
Pork Clitub Diels
Figure 19: Annotation platform Pee et al. 2022) for collecting human evaluation of expert verbal feedback.
38

---

Published as a conference paper at ICLR 2025
1. Configurationn File Reading**; The "previous_code correctly reads the configuration file using "cenfigparser', However, ensure that the configuration file path is valid and exists before attempting to read it. This is not explicitly checked in the "previcus_code"
2
2. **Directory Existence Check**: The "previous_come uses "os. cos.path.existe (project_air)) "ir) to check it the project directory exists. While this works, "os. "os.path.isdin (project_dir)" to specifically check for directory existence, as it is more semantically correct.
more appropriate
3. *IIIP Archive Creation*:: The 'previous code" attempts create the ZIP archive using "shutil masts..make_eee make_archive (project (projet dir, 'zip', archive_dir)", This is incorrect because "shutil.make_archi
expects the base name of the archive and the root directory to archive. The correct usage should be @hhutil.aak_aarchine
sptiittttttttttttttttt.eeeeeett. splitext (zip_file_pas [0], format-'zip', root_árrpprojcc__din"
4. Exception Handlinge:: The "previcus code" raises generic "Exception" if the ZIP archive creation fails. While this is acceptable, it is better to previde a more specific error message indicating the failure reason. Additionally, ensure that the ZIP file is actually created by checking its existence after the sahtiilmake_archive call.
5. Retuurn Value The pprevious_code correctly returns "True' if the ZIP archive is successfully created. Hovever, it should also ensure that the ZIP file exists befere returning "True".
6. Code Simplicity and Readability+:: The 'previous_code includes detailed docstring, which is good practice. However, the actual impl empllmmetaiion can be simplified and made more readable by following ng the correct usage of "shutil.make__d""" makee_rrchive" and ensuring proper exception handling.
13 Overall, the "previous correctiens in the directory existence check, ZIP archive creation, and exception handling to function correctly.
"odde code" has the right structure but needs
Figure 20: example case that the feedback guides toward the ground truth without explicitly referencing it.
39

---

Published as a conference paper at ICLR 2025
1. Class Name+*; The class name in the "previous_code is Email Maanlee',err', but it should be "maillRepuestaaddee to match the "round_truuth_cdd".
2. *oontenttType Check Instead of directly checking the header, use "ogi. header" to parse the header and then check if "ctype != "application//som".
**Error Handling for Content-Type*::
the "Content-Type" is not 'application/json", simply send 400 response and end headers without writing a message to the response body.
4. Reading Content-Lengthi: Use "length int int
get content-leegthh") instead of "content_lengte "Content-Length! 0))*
5. **J9ON Decoding++: When catching "json. Joo send 400 response and end headers without writing message to the response body.
10
6. ..issinng Fields Check when required fields are missing, send 400 response and end headers without writing message to the response body.
12
7. **TM authentication Error Handling**; When catching "smtplib.sMI www.eeauubeniiattt.tozrro.o. send 535 response and end headers without writing a message to the response body.
15 8. G*General Exception Handlings indslnngx: Remove the general exception handler that sends 500 response, as it is not present in the "rrunna_tuuttti trunu
17 By making these changes, the porevicus_code will align more closely with the ground_truth_code".
Figure 21: An example that the feedback directly references ground truth, leading to "leakage".
40


Reviews:
[{"review_id": "y7PwV5IKyF", "reviewer": "Reviewer_cHoy", "summary": "The authors introduce ConvCodeWorld, a novel framework designed to benchmark the code generation performance of large language models (LLMs) in a conversational context. This setting allows LLMs to receive various types of feedback, including compiler messages, execution outputs, and human-like verbal feedback. To support reproducibility, the authors developed a benchmark, ConvCodeBench, that uses ConvCodeWorld with pre-generated feedback logs. They conducted a comprehensive evaluation of state-of-the-art (SOTA) LLMs and a detailed ablation study, providing valuable insights to inform future research on conversational code generation.", "questions": "- While the authors evaluate verbal feedback with proprietary GPT models, it would be beneficial to include SOTA open-weight models as well. Given that GPT models evolve frequently, this reliance may risk data leakage. Testing open-weight models, even if currently less performant, could strengthen the reproducibility and discussion of findings.\n- The claim in Appendix A.2 that the approach achieves \"1.5% of the cost of human annotation\" seems optimistic. Beyond the token generation cost, quality and accuracy of the model-generated content should also be factored in.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The primary strength of this paper lies in its comprehensive approach:\n\n- The authors cover a broad spectrum of models, including proprietary and open-weight LLMs.\n- They thoroughly explore feedback settings across multiple types—compilation, execution, and verbal.\n- Extensive ablation studies and analyses reveal findings that highlight key factors for advancing conversational coding tasks.", "weaknesses": "- **Novelty:** While this paper offers valuable insights into evaluating conversational code generation, the topic itself is not new. Prior works like InterCode and MINT have explored compilation, execution, and verbal feedback mechanisms (see Table 6).\n- **Clarity of Writing:** The paper’s clarity could be improved, particularly in the introduction. Expanding the background on conversational code generation tasks and typical settings would help readers appreciate the unique contributions of this work. Additionally, certain terminology, like \"partial coverage\" in Table 1, could be more explicitly explained.\n- **Experiment Details:** Some important experimental details are missing or unclear:\n    1. **Temperature settings** for experiments are not specified. This detail is relevant as multi-turn interactions could benefit from more varied output.\n    2. The **construction of the proposed dataset** is insufficiently explained, particularly given the reported 29% success rate on BigCodeBench. This contrasts with single-turn performance on ConvCodeWorld, which appears higher.\n    3. **Prompting methods** lack clear definition. Both Section 4.1 and Appendix B are vague on this, with no explicit description of the prompt setup for experiments. \n    4. The construction of $f_v$ and $f_v^*$ is not well explained. Examples in Section 2.1.1 and Appendix F could be enhanced with a more formal, detailed breakdown of these prompt components.\n     5. In Section A.3, could the authors clarify the meaning of \"a model referenced ground truth code in $f_v^*$\"? Why would the model reference ground truth, and could examples illustrate this behavior?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GqWA6iRuJs", "reviewer": "Reviewer_3Fw3", "summary": "This paper introduces ConvCodeWorld, a benchmark to evaluate LLMs in solving programming problems via multi-turn interactions with diverse feedback combinations. The benchmark encompasses 9 distinct feedback scenarios that combine compilation feedback, execution feedback with varying test coverage, and real-time verbal feedback simulated at different expertise levels using GPT-4o.  Additionally, a static version called ConvCodeBench is proposed, which uses pre-generated feedback logs to reduce computational costs while maintaining a high correlation with the live benchmark. In experiments, a comprehensive set of open/closed-source LLMs are evaluated on ConvCodeWorld and ConvCodeBench.", "questions": "1. Could the author add more comparisons between ConvCodeWorld and MINT to enhance the clarity?\n2. Could the author address the discrepancy in feedback simulation compared to actual human feedback?\n3. Could the author explain the evaluation setup with partial execution feedback included? A follow-up question would be what if the problem does not have full coverage tests? How many of such cases exist in the dataset?\n4. Could the author respond to the potential bias in the static benchmark?  A better way to show the correlation between ConvCodeWorld and ConvCodeBench might involve presenting the error patterns between the reference model and the target model.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Evaluating LLMs in interactive environments is a significant and emerging area of study.\n2. The diverse combination of different feedback is reasonable.\n3. The introduction of static alternative ConvCodeBench reduces the overhead of real-time feedback generation making it scalable and practical for large-scale experiments.\n4. The authors conduct comprehensive experiments covering 17 open- and closed-source LLMs.\n5. The paper is easy-to-follow.", "weaknesses": "1. In the MINT paper (Wang et al., 2024), the multi-turn interaction (Figure 1, page 3) already includes execution results and human feedback. The human feedback in MINT covered both novice feedback (referred to as \"lazy user\" feedback in MINT) and expert feedback (referred to as \"natural language\" feedback in MINT). Given this, the novelty of this paper appears limited. Could the author include more comparisons between ConvCodeWorld and MINT to enhance clarity and novelty?\n    - Reference: Wang, X., Wang, Z., Liu, J., Chen, Y., Yuan, L., Peng, H., & Ji, H. MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback. In The Twelfth International Conference on Learning Representations.\n\n2. Discrepancy in feedback simulation compared to actual human feedback\n    - Figure 13 (appendix) shows one example of the simulated expert-level verbal feedback, which is overly detailed and well-structured. I am concerned that it might not accurately reflect how real experts provide feedback when using LLMs in coding. An analysis comparing simulated feedback to actual human feedback would be both helpful and necessary.\n\n    - A similar concern was found in the simulated novice-level verbal feedback (Figure 12, appendix). The simulated novice-level feedback includes information that seems unrealistic for novice programmers (e.g., \"Also, I think there exists a simpler way to sort a list in Python.\"). How could a novice programmer provide such feedback based on the compilation and execution results? As suggested above, an analysis of how the simulated feedback relates to actual human feedback is necessary.\n\n3. Lack of Clarity in Evaluation Setup: The process of using partial execution feedback was not clearly explained. Did the author randomly omit test cases from the list of tests with full coverage? If so, what criteria were applied to ensure that the selection of test cases was reasonable? Providing a detailed experimental setup would enhance clarity.\n\n4. The claim that \"Weaker Models with Feedback Surpassing Single-Turn SOTA Models\" seems an unfair comparison.  While I understand that the aim is to highlight the value of multi-turn interactions, weaker models with multi-turn feedback inherently benefit from additional input context compared to single-turn, stronger models. It is therefore unsurprising that the weaker model with multi-turn interactions outperforms single-turn, stronger models. A fair comparison would evaluate the same LLMs in both single-turn and multi-turn scenarios (I thought this could also be concluded from the results section). The statement about using the same LLM would more effectively emphasize the role of multi-turn interactions in enhancing performance.\n\n5. Bais in Static ConvCodeBench\n    - The use of pre-generated feedback logs from a reference model seems biased. The errors made by the reference model might not reflect the actual errors the target model would encounter on the same problem. Thus, successful error correction might not be attributed to the feedback (or multi-turn interaction) on the code generated by the reference model, as the target model may already be capable of solving the problem on its own.\n    - The correlation calculated between ConvCodeWorld and ConvCodeBench is unreasonable based on the MRR and Recall since the results may be affected by the bias mentioned above. A better way to show this correlation is to show the correlation of error patterns existing in the generated code between the reference model and the target model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GMkY9VYpAp", "reviewer": "Reviewer_gBAk", "summary": "The authors propose ConvCodeWorld, an environment for evaluating multi-turn code generation across 9 diverse feedback combinations. This entails compilation feedback, execution feedback (partial and full test coverage), and LLM-simulated verbal feedback (novice-level and expert-level). For this, they transform BigCodeBench-Instruct, a single-turn Python code generation benchmark. Additionally, they also propose a static benchmark to serve more practical purposes, ConvCodeBench, with pre-computed logs based on a weak model (CodeLlama-7B-Instruct). They find that performance on ConvCodeBench strongly correlates with performance on ConvCodeWorld. Using ConvCodeWorld, they evaluate 17 (3 closed-source and 14 open-source) LLMs of varying sizes. Through this, they highlight many key findings, including: (1)  Performance varies across feedback settings, often affecting the rankings, highlighting the need to select models based on the feedback available, (2) Weaker models with feedback can surpass single-turn SOTA models, (3) Models trained on a specific type of feedback struggle to generalize to unseen combinations of feedback, and (4) There is a tradeoff between an LLM's ability to solve problems in fewer turns (higher MRR) and solve many problems in total (high recall).", "questions": "- Could you clarify that the number of samples for the single-turn baseline (i.e., first column in Table 3 and Table 4) is the same as the number of maximum turns?\n- Currently, it seems that for a given example, the same combination of feedback is given at each turn. Have you considered varying the type of feedback at different turns (e.g., compiler only in turn 1, compiler and execution in turn 2, compiler and novice feedback in turn 3, compiler and expert feedback in turn 4). Perhaps certain types of feedback or more useful (and more practical) at different steps?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- As the authors have highlighted, evaluating multi-turn code generation is very difficult. By providing an environment with access to many diverse combinations of feedback, ConvCodeWorld could potentially be very useful to the research community, to hill-climb on multi-turn coding capabilities of LLMs.\n- The authors have been very thorough about exploring different types of scenarios, with partial/full execution feedback, and novice/expert-level verbal feedback. This allows more fine-grained evaluation.\n- There is an extensive analysis of performance of many different LLMs, and the key findings will likely be very interesting to the research community.", "weaknesses": "- While I agree that having a static benchmark like ConvCodeBench is useful, I am not convinced that it can be interpreted as a multi-turn evaluation set. Namely, my understanding is that it does not evaluate whether a model can iteratively improve its own prediction. The output from the previous turn is pre-defined, and so this seems to simply be trying to repair a program (which the model under test did not generate). If it fails to do it at a given turn, the output of that turn is ignored, and in the next turn, a fixed program is provided, which could be very different from the program the model under test generated in the previous turn. This setting seems to be a bit unnatural to me.\n- The environment is constrained to Python only. \n\n\nSuggestion:\n- It would have been interesting to have an ablation without the compiler feedback.\n- Other relevant paper to cite: https://arxiv.org/pdf/2306.09896", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Efabf6LsBA", "reviewer": "Reviewer_acNZ", "summary": "This work presents ConvCodeWorld, a benchmark for evaluating how well language models use feedback to address a code generation problem across multiple turns. While multi-turn interactive code generation work exists, the impact of the type of feedback on model performance is not as thoroughly examined. To carry out this study, the authors formulate three types of feedback (compilation, execution, verbal) and construct a pipeline (ConvCodeBench) that bootstraps existing code generation benchmarks to construct a static benchmark that follows the ConvCodeWorld formulation. In the results, the authors find that generally, closed source models perform best. Through ablations of different feedback techniques and open source models, the authors also discuss trends around the topics of generalization, the impact of turns, and the correlation between ConvCodeBench and ConvCodeWorld.", "questions": "- How are you detecting whether ground truth code is leaked by the GPT 4o based expert? From reading A.3, I couldn’t quite understand this. Is the `ground_truth_code` function doing an exact match lookup in GPT 4o’s feedback? Or is there another implementation? Table 7 seems to be justifying that the verbal feedback doesn’t leak ground truth code because of downstream task performance differences, instead of a direct examination of the verbal feedback’s content.\n- What does Table 9 mean? When it is first referenced in Section 2.1.1, the phi character does not appear to be defined at that point in the paper. Also, GPT 4/4o models are tested exclusively. Any reason why Claude / Llama / other models were not used?\n- I agree with the justification that human annotators are more costly than GPT 4o. However, is it possible that novice/expert-level feedback looks very different between humans and language models? I think a justification for why LMs are a good proxy for human feedback for ConvCodeWorld would be good assurance.\n- Line 247: Which two settings are being referenced for this comparison?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Feedback in code generation feels like a fairly understudied area. The premise of the paper is quite interesting in my opinion. The results are also neat, particularly how different models perform well under different feedback scenarios.\n- The fact that ConvCodeBench bootstraps existing code generation benchmarks is quite compelling. While the authors apply this towards BigCodeBench, it sounds like it would work for a wide variety of similar benchmarks.\n- The evaluation is quite comprehensive, and the experimental setup is easy to understand + sound.\n- Section 4.2.3 - Given the knowledge about how ReflectionCoder was trained, it is quite interesting to see that the model’s performance varies with the type of feedback provided.\n- The authors’ experiments reveal a number of interesting insights. I particularly liked how ConvCodeBench performance is shown to be a good proxy for ConvCodeWorld performance, and the relationship between MRR and recall.", "weaknesses": "- Section 2.1.1 is written in a way that a lot of references are made to the appendix, which requires a lot of jumping back and forth. As Appendix A.3 is referenced multiple times, it might be worth considering just putting the information justifying the claims discussed here directly in the main paper.\n- The mathematical notation in the paper is formalized correctly. From a readability standpoint, I would have preferred to just read what feedback types correspond to what results, as opposed to having to map symbols back to their corresponding feedback. Section 4.2.1 was a bit difficult to parse because of this. Of course, this is just my opinion.\n- For Takeaway 2 (Impact of Expert-level…) in Section 4.2.1 - It was helpful to be told these observations as opposed to reading them from the table, but an explanation for why such trends were observed feels a bit lacking (“struggle to utilize complex feedback” feels a bit coarse-grained as a justification for a lot of observations)\n- I’m not sure the takeaway – that feedback helps weaker models perform stronger than stronger models in a zero-shot setting – is that surprising given prior work (e.g. Mint, InterCode discuss this to some capacity).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)