ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments

Authors: Hojae Han, seung-won hwang, Rajhans Samdani, Yuxiong He
Primary area: datasets and benchmarks
Keywords: Large language models, Multi-turn code generation, Benchmark

Abstract:
Large language models (LLMs) have proven invaluable for code generation, particularly in interactive settings. However, existing code generation benchmarks fail to capture the diverse feedback encountered in multi-turn interactions, limiting our ability to evaluate LLMs in these contexts. To address this gap, we present a set of novel benchmarks that explicitly model the quality of feedback provided to code generation LLMs. Our contributions are threefold: **First**, we introduce CONVCODEWORLD, a novel and reproducible environment for benchmarking interactive code generation. CONVCODEWORLD simulates 9 distinct interactive code generation scenarios while systematically combining three types of feedback: (a) compilation feedback; (b) execution feedback with varying test coverage; (c) verbal feedback generated by GPT-4o with different levels of expertise. **Second**, we introduce CONVCODEBENCH, a fast, static version of benchmark that uses pre-generated feedback logs, eliminating the need for costly dynamic verbal feedback generation while maintaining
strong Spearman’s rank correlations (0.82 to 0.99) with CONVCODEWORLD. **Third**, extensive evaluations of both closed-source and open-source LLMs including R1-Distill on CONVCODEWORLD reveal key insights: (a) LLM performance varies significantly based on the feedback provided; (b) Weaker LLMs, with sufficient feedback, can outperform single-turn results of state-of-the-art LLMs without feedback; (c) Training on a specific feedback combination can limit an LLM’s ability to utilize unseen combinations; (d) LLMs solve problems in fewer turns (high MRR) may not solve as many problems overall (high Recall), and vice versa. All implementations and benchmarks will be made publicly available at https://huggingface.co/spaces/ConvCodeWorld/ConvCodeWorld

Decision: accept (poster)
Reviewer scores: 5, 5, 8, 6

Reviews and discussion:
<thread>
creation date:
1729561586748

thread_type:
official_review

<thread_object>
creator: reviewer_acnz

<content>
summary:
This work presents ConvCodeWorld, a benchmark for evaluating how well language models use feedback to address a code generation problem across multiple turns. While multi-turn interactive code generation work exists, the impact of the type of feedback on model performance is not as thoroughly examined. To carry out this study, the authors formulate three types of feedback (compilation, execution, verbal) and construct a pipeline (ConvCodeBench) that bootstraps existing code generation benchmarks to construct a static benchmark that follows the ConvCodeWorld formulation. In the results, the authors find that generally, closed source models perform best. Through ablations of different feedback techniques and open source models, the authors also discuss trends around the topics of generalization, the impact of turns, and the correlation between ConvCodeBench and ConvCodeWorld.

strengths:
- Feedback in code generation feels like a fairly understudied area. The premise of the paper is quite interesting in my opinion. The results are also neat, particularly how different models perform well under different feedback scenarios.
- The fact that ConvCodeBench bootstraps existing code generation benchmarks is quite compelling. While the authors apply this towards BigCodeBench, it sounds like it would work for a wide variety of similar benchmarks.
- The evaluation is quite comprehensive, and the experimental setup is easy to understand + sound.
- Section 4.2.3 - Given the knowledge about how ReflectionCoder was trained, it is quite interesting to see that the model’s performance varies with the type of feedback provided.
- The authors’ experiments reveal a number of interesting insights. I particularly liked how ConvCodeBench performance is shown to be a good proxy for ConvCodeWorld performance, and the relationship between MRR and recall.

weaknesses:
- Section 2.1.1 is written in a way that a lot of references are made to the appendix, which requires a lot of jumping back and forth. As Appendix A.3 is referenced multiple times, it might be worth considering just putting the information justifying the claims discussed here directly in the main paper.
- The mathematical notation in the paper is formalized correctly. From a readability standpoint, I would have preferred to just read what feedback types correspond to what results, as opposed to having to map symbols back to their corresponding feedback. Section 4.2.1 was a bit difficult to parse because of this. Of course, this is just my opinion.
- For Takeaway 2 (Impact of Expert-level…) in Section 4.2.1 - It was helpful to be told these observations as opposed to reading them from the table, but an explanation for why such trends were observed feels a bit lacking (“struggle to utilize complex feedback” feels a bit coarse-grained as a justification for a lot of observations)
- I’m not sure the takeaway – that feedback helps weaker models perform stronger than stronger models in a zero-shot setting – is that surprising given prior work (e.g. Mint, InterCode discuss this to some capacity).

questions:
- How are you detecting whether ground truth code is leaked by the GPT 4o based expert? From reading A.3, I couldn’t quite understand this. Is the `ground_truth_code` function doing an exact match lookup in GPT 4o’s feedback? Or is there another implementation? Table 7 seems to be justifying that the verbal feedback doesn’t leak ground truth code because of downstream task performance differences, instead of a direct examination of the verbal feedback’s content.
- What does Table 9 mean? When it is first referenced in Section 2.1.1, the phi character does not appear to be defined at that point in the paper. Also, GPT 4/4o models are tested exclusively. Any reason why Claude / Llama / other models were not used?
- I agree with the justification that human annotators are more costly than GPT 4o. However, is it possible that novice/expert-level feedback looks very different between humans and language models? I think a justification for why LMs are a good proxy for human feedback for ConvCodeWorld would be good assurance.
- Line 247: Which two settings are being referenced for this comparison?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729715615379

thread_type:
official_review

<thread_object>
creator: reviewer_gbak

<content>
summary:
The authors propose ConvCodeWorld, an environment for evaluating multi-turn code generation across 9 diverse feedback combinations. This entails compilation feedback, execution feedback (partial and full test coverage), and LLM-simulated verbal feedback (novice-level and expert-level). For this, they transform BigCodeBench-Instruct, a single-turn Python code generation benchmark. Additionally, they also propose a static benchmark to serve more practical purposes, ConvCodeBench, with pre-computed logs based on a weak model (CodeLlama-7B-Instruct). They find that performance on ConvCodeBench strongly correlates with performance on ConvCodeWorld. Using ConvCodeWorld, they evaluate 17 (3 closed-source and 14 open-source) LLMs of varying sizes. Through this, they highlight many key findings, including: (1)  Performance varies across feedback settings, often affecting the rankings, highlighting the need to select models based on the feedback available, (2) Weaker models with feedback can surpass single-turn SOTA models, (3) Models trained on a specific type of feedback struggle to generalize to unseen combinations of feedback, and (4) There is a tradeoff between an LLM's ability to solve problems in fewer turns (higher MRR) and solve many problems in total (high recall).

strengths:
- As the authors have highlighted, evaluating multi-turn code generation is very difficult. By providing an environment with access to many diverse combinations of feedback, ConvCodeWorld could potentially be very useful to the research community, to hill-climb on multi-turn coding capabilities of LLMs.
- The authors have been very thorough about exploring different types of scenarios, with partial/full execution feedback, and novice/expert-level verbal feedback. This allows more fine-grained evaluation.
- There is an extensive analysis of performance of many different LLMs, and the key findings will likely be very interesting to the research community.

weaknesses:
- While I agree that having a static benchmark like ConvCodeBench is useful, I am not convinced that it can be interpreted as a multi-turn evaluation set. Namely, my understanding is that it does not evaluate whether a model can iteratively improve its own prediction. The output from the previous turn is pre-defined, and so this seems to simply be trying to repair a program (which the model under test did not generate). If it fails to do it at a given turn, the output of that turn is ignored, and in the next turn, a fixed program is provided, which could be very different from the program the model under test generated in the previous turn. This setting seems to be a bit unnatural to me.
- The environment is constrained to Python only. 


Suggestion:
- It would have been interesting to have an ablation without the compiler feedback.
- Other relevant paper to cite: https://arxiv.org/pdf/2306.09896

questions:
- Could you clarify that the number of samples for the single-turn baseline (i.e., first column in Table 3 and Table 4) is the same as the number of maximum turns?
- Currently, it seems that for a given example, the same combination of feedback is given at each turn. Have you considered varying the type of feedback at different turns (e.g., compiler only in turn 1, compiler and execution in turn 2, compiler and novice feedback in turn 3, compiler and expert feedback in turn 4). Perhaps certain types of feedback or more useful (and more practical) at different steps?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729878929267

thread_type:
official_review

<thread_object>
creator: reviewer_3fw3

<content>
summary:
This paper introduces ConvCodeWorld, a benchmark to evaluate LLMs in solving programming problems via multi-turn interactions with diverse feedback combinations. The benchmark encompasses 9 distinct feedback scenarios that combine compilation feedback, execution feedback with varying test coverage, and real-time verbal feedback simulated at different expertise levels using GPT-4o.  Additionally, a static version called ConvCodeBench is proposed, which uses pre-generated feedback logs to reduce computational costs while maintaining a high correlation with the live benchmark. In experiments, a comprehensive set of open/closed-source LLMs are evaluated on ConvCodeWorld and ConvCodeBench.

strengths:
1. Evaluating LLMs in interactive environments is a significant and emerging area of study.
2. The diverse combination of different feedback is reasonable.
3. The introduction of static alternative ConvCodeBench reduces the overhead of real-time feedback generation making it scalable and practical for large-scale experiments.
4. The authors conduct comprehensive experiments covering 17 open- and closed-source LLMs.
5. The paper is easy-to-follow.

weaknesses:
1. In the MINT paper (Wang et al., 2024), the multi-turn interaction (Figure 1, page 3) already includes execution results and human feedback. The human feedback in MINT covered both novice feedback (referred to as "lazy user" feedback in MINT) and expert feedback (referred to as "natural language" feedback in MINT). Given this, the novelty of this paper appears limited. Could the author include more comparisons between ConvCodeWorld and MINT to enhance clarity and novelty?
    - Reference: Wang, X., Wang, Z., Liu, J., Chen, Y., Yuan, L., Peng, H., & Ji, H. MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback. In The Twelfth International Conference on Learning Representations.

2. Discrepancy in feedback simulation compared to actual human feedback
    - Figure 13 (appendix) shows one example of the simulated expert-level verbal feedback, which is overly detailed and well-structured. I am concerned that it might not accurately reflect how real experts provide feedback when using LLMs in coding. An analysis comparing simulated feedback to actual human feedback would be both helpful and necessary.

    - A similar concern was found in the simulated novice-level verbal feedback (Figure 12, appendix). The simulated novice-level feedback includes information that seems unrealistic for novice programmers (e.g., "Also, I think there exists a simpler way to sort a list in Python."). How could a novice programmer provide such feedback based on the compilation and execution results? As suggested above, an analysis of how the simulated feedback relates to actual human feedback is necessary.

3. Lack of Clarity in Evaluation Setup: The process of using partial execution feedback was not clearly explained. Did the author randomly omit test cases from the list of tests with full coverage? If so, what criteria were applied to ensure that the selection of test cases was reasonable? Providing a detailed experimental setup would enhance clarity.

4. The claim that "Weaker Models with Feedback Surpassing Single-Turn SOTA Models" seems an unfair comparison.  While I understand that the aim is to highlight the value of multi-turn interactions, weaker models with multi-turn feedback inherently benefit from additional input context compared to single-turn, stronger models. It is therefore unsurprising that the weaker model with multi-turn interactions outperforms single-turn, stronger models. A fair comparison would evaluate the same LLMs in both single-turn and multi-turn scenarios (I thought this could also be concluded from the results section). The statement about using the same LLM would more effectively emphasize the role of multi-turn interactions in enhancing performance.

5. Bais in Static ConvCodeBench
    - The use of pre-generated feedback logs from a reference model seems biased. The errors made by the reference model might not reflect the actual errors the target model would encounter on the same problem. Thus, successful error correction might not be attributed to the feedback (or multi-turn interaction) on the code generated by the reference model, as the target model may already be capable of solving the problem on its own.
    - The correlation calculated between ConvCodeWorld and ConvCodeBench is unreasonable based on the MRR and Recall since the results may be affected by the bias mentioned above. A better way to show this correlation is to show the correlation of error patterns existing in the generated code between the reference model and the target model.

questions:
1. Could the author add more comparisons between ConvCodeWorld and MINT to enhance the clarity?
2. Could the author address the discrepancy in feedback simulation compared to actual human feedback?
3. Could the author explain the evaluation setup with partial execution feedback included? A follow-up question would be what if the problem does not have full coverage tests? How many of such cases exist in the dataset?
4. Could the author respond to the potential bias in the static benchmark?  A better way to show the correlation between ConvCodeWorld and ConvCodeBench might involve presenting the error patterns between the reference model and the target model.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716739980

thread_type:
official_review

<thread_object>
creator: reviewer_choy

<content>
summary:
The authors introduce ConvCodeWorld, a novel framework designed to benchmark the code generation performance of large language models (LLMs) in a conversational context. This setting allows LLMs to receive various types of feedback, including compiler messages, execution outputs, and human-like verbal feedback. To support reproducibility, the authors developed a benchmark, ConvCodeBench, that uses ConvCodeWorld with pre-generated feedback logs. They conducted a comprehensive evaluation of state-of-the-art (SOTA) LLMs and a detailed ablation study, providing valuable insights to inform future research on conversational code generation.

strengths:
The primary strength of this paper lies in its comprehensive approach:

- The authors cover a broad spectrum of models, including proprietary and open-weight LLMs.
- They thoroughly explore feedback settings across multiple types—compilation, execution, and verbal.
- Extensive ablation studies and analyses reveal findings that highlight key factors for advancing conversational coding tasks.

weaknesses:
- **Novelty:** While this paper offers valuable insights into evaluating conversational code generation, the topic itself is not new. Prior works like InterCode and MINT have explored compilation, execution, and verbal feedback mechanisms (see Table 6).
- **Clarity of Writing:** The paper’s clarity could be improved, particularly in the introduction. Expanding the background on conversational code generation tasks and typical settings would help readers appreciate the unique contributions of this work. Additionally, certain terminology, like "partial coverage" in Table 1, could be more explicitly explained.
- **Experiment Details:** Some important experimental details are missing or unclear:
    1. **Temperature settings** for experiments are not specified. This detail is relevant as multi-turn interactions could benefit from more varied output.
    2. The **construction of the proposed dataset** is insufficiently explained, particularly given the reported 29% success rate on BigCodeBench. This contrasts with single-turn performance on ConvCodeWorld, which appears higher.
    3. **Prompting methods** lack clear definition. Both Section 4.1 and Appendix B are vague on this, with no explicit description of the prompt setup for experiments. 
    4. The construction of $f_v$ and $f_v^*$ is not well explained. Examples in Section 2.1.1 and Appendix F could be enhanced with a more formal, detailed breakdown of these prompt components.
     5. In Section A.3, could the authors clarify the meaning of "a model referenced ground truth code in $f_v^*$"? Why would the model reference ground truth, and could examples illustrate this behavior?

questions:
- While the authors evaluate verbal feedback with proprietary GPT models, it would be beneficial to include SOTA open-weight models as well. Given that GPT models evolve frequently, this reliance may risk data leakage. Testing open-weight models, even if currently less performant, could strengthen the reproducibility and discussion of findings.
- The claim in Appendix A.2 that the approach achieves "1.5% of the cost of human annotation" seems optimistic. Beyond the token generation cost, quality and accuracy of the model-generated content should also be factored in.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces ConvCodeWorld, a benchmark designed to assess the ability of large language models (LLMs) to solve programming problems through multi-turn interactions featuring diverse feedback mechanisms. The benchmark includes nine distinct feedback scenarios, combining compilation feedback, execution feedback with varying test coverage, and simulated real-time verbal feedback modeled at different expertise levels using GPT-4. To complement this, the authors propose a static counterpart, ConvCodeBench, which employs pre-generated feedback logs to minimize computational overhead while maintaining a strong correlation with the live benchmark.

The study evaluates a broad spectrum of open- and closed-source LLMs on both ConvCodeWorld and ConvCodeBench, offering comprehensive experimental results. While conversational approaches to code generation are widely acknowledged, benchmarks in this area remain relatively underdeveloped. This work contributes a valuable resource for the community to explore this crucial problem, delivering meaningful insights through its thorough and diverse evaluations.