EXecution-Eval: Can language models execute real-world code?

Authors: Rob Kopel
Primary area: datasets and benchmarks
Keywords: large language model, evaluation, benchmark, code execution

Abstract:
As Large Language Models (LLMs) advance, traditional benchmarks face challenges of dataset saturation and disconnection from real-world performance, limiting our understanding of true model capabilities. We introduce EXecution-Eval (EXE), a benchmark designed to assess LLMs' ability to execute code and predict program states. EXE attempts to address key limitations in existing evaluations: difficulty scaling, task diversity, training data contamination, and cost-effective scalability.
Comprising over 30,000 tasks derived from 1,000 popular Python repositories on GitHub, EXE spans a range of context lengths and algorithmic complexities. Tasks require models to execute code, necessitating various operations including mathematical reasoning, logical inference, bit manipulation, string operations, loop execution, and maintaining multiple internal variable states during computation. Our methodology involves: (a) selecting and preprocessing GitHub repositories, (b) generating diverse inputs for functions, (c) executing code to obtain ground truth outputs, and (d) formulating tasks that require models to reason about code execution. This approach allows for continuous new task generation for as few as 1,200 tokens, significantly reducing the risk of models "training on the test set."
We evaluate several state-of-the-art LLMs on EXE, revealing insights into their code comprehension and execution capabilities. Our results show that even the best-performing models struggle with complex, multi-step execution tasks, highlighting specific computational concepts that pose the greatest challenges for today's LLMs. Furthermore, we review EXE's potential for finding and predicting errors to aid in assessing a model's cybersecurity capabilities. We propose EXE as a sustainable and challenging testbed for evaluating frontier models, offering potential insights into their internal mechanistic advancement

Decision: reject
Reviewer scores: 3, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1729356976402

thread_type:
official_review

<thread_object>
creator: reviewer_o62o

<content>
summary:
This paper introduces a new benchmark EXE, focusing on testing the capability of LLMs to simulate code execution. EXE is made up of over 30000 tasks derived from 1,000 popular Python repositories on GitHub. In this scenario, LLMs need to execute code, involving operations like mathematical reasoning, logical inference, loop execution, and maintaining internal variable states. This paper provides a shallow breakdown on this. The pipeline to create EXE involves selecting and preprocessing GitHub repositories, synthesizing inputs based on function signatures, and then creating test cases (unit tests, and potentially, chaining functions tests) with the inputs. The authors claim their pipeline is automatic and capable of continuous new task generation with newest repositories to avoid test set contamination.


————after rebuttal————

The substantial revisions made during the rebuttal period addressed some of my concerns regarding model evaluation and test set contamination. The clarifications on dependency solving are also convincing. However, as the rest of my concerns are not fully addressed and the substantial revisions are making it a new paper with too many raw details without careful organizations. I think it would be better to be revised and submitted to follow-up venues like ICML. Besides, knowing the capabilities of LLMs in code executions should be the outcome of this paper, and I think current easy subset(see weakness 8) evaluation somehow weakens it.

Nonetheless, I have raised my score to 5 and presentation to 2 to praise for the authors’ efforts.

Good luck,

Reviewer o62o

strengths:
1.Provide a benchmark of real-world Python code for testing LLM execution, the test cases are significantly harder and more representative for real-world usage, therefore providing a more realistic assessment of model capabilities, 

2.Establish an automatic pipeline to create a real-world dataset for LLM-based code execution tasks.

3.Cover a wide range of programming concepts and can be potentially scaled up or updated with new tasks.

4.The unit-test based evaluation is correct, the authors also mention the potential to create more complicated test cases like using chaining functions.

weaknesses:
## Major weaknesses:
1.Only GPT-4o and GPT-4o-mini are evaluated, contrary to the claim of evaluating **"several state-of-the-art LLMs."** Additional evaluation with different LLMs are recommended, like Claude, Gemini, Deepseek, Phi, Qwen, etc.

2.The claim of **"avoiding training on the test set"** relies heavily on the quality and effectiveness of the pipeline's ability to generate new test cases, which is not thoroughly demonstrated in the paper, no supplementary materials provided either. The Lack of supportive materials (either the benchmark itself or its creating code) to support claims about the framework's capabilities, weakens the contribution of a dataset paper.

3.The handling of import dependencies and the process of inlining required elements are not clearly explained. It's technically important here. Need clarification.

4.A bit limited to Python code, which may not represent the full spectrum of programming challenges across different languages. Since LLMs are pretrained on various programming languages, it's worth to know the execution capability on other programming languages.

5.Poor quality of figures in the paper, with low-precision images that are difficult to see clearly, the authors should use vector figures instead of jpgs or pngs, 

6.The appendix uses 8 pages to show an example, which is excessive and poorly organized, besides, it's still not intuitive for understanding. This needs significant revision for clarity and conciseness.

## Minor weaknesses:

7.A bit limited evaluation metrics, using only Pass@1 accuracy. Considering more evaluations on Pass@k, or try some self-correction mechanism with LLM.

8.Filtering on limited acceptable types and functions seems to make EXE an **easy subset of the real real-world programs**, although it is a fair design choice for a benchmark to avoid environment configuration issues. I think it's more interesting to know the capabilities and limitations of LLMs when executing harder cases, containing real-world types like numpy.array, torch.tensor for example. Can the authors add some discussions about their findings here?

## Typos and Presentation Issues:

Line 294: tense issues, ...**increase** task difficulty, however bit manipulation and boolean operations only **showed**...  Should use unified tense throughout a paragraph.

Line 297: however for loops **on (73 Pass@1) on** average did not have a significant impact.

Line 303: Incorrect spacing on the title of the rightmost subfigure.

Figure 7: Examining only on LLM really executed code makes the accuracy normal now. However, it seems the results are not clearly illsutrated (only a small part of the figure is valid now, which is not clear). Consider to use some new figures.

Appendix A.2: These are important part of your paper, since current version only uses 8 pages, consider to move this section to the main page and explain them with more details.

questions:
1.In the appendix, could you clearly differentiate between original code, imported dependencies, and LLM execution steps? Can you show the full LLM output and indicate at which steps they fail?

2.How does EXE compare to existing code execution benchmarks in terms of task diversity and difficulty when "executing code"?

3.Can you elaborate on the measures taken to ensure the generated test cases are meaningful, diverse, and correctly assess code execution abilities?

4.How do you validate that the newly generated test cases are indeed novel and not present in existing LLM training sets?

5.Has the chaining-function been implemented now? Because i think it will be of more interests to the community if EXE can create more complicated test cases automatically.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730224271509

thread_type:
official_review

<thread_object>
creator: reviewer_vhj6

<content>
summary:
The authors introduce a dataset of executable python functions mined from Github. The functions chosen have certain type annotations for which test cases can be generated. The task consists in providing a code snippet as well as the input arguments into an LLM and asking the LLM to predict the output (this task has been referred to as "program induction" in some literature of the past, and I will refer to it as "program understanding")

The authors argue that this is a non-trivial benchmark and that the methodology allows the benchmark to evolve over time to include test cases or functions that are not in the training set. The authors also argue that this program understanding task could be an useful gauge of LLMs performance for coding tasks. 

The authors evaluate GPT4o and GPT4o-mini on this task and provide some analysis on performance by certain proxies for ``difficulty" such as lines of code, number of function calls, etc.

strengths:
The authors deserve credit for their creative use of open-source software on Github. I believe that more executable coding benchmarks will be beneficial to the community and the authors have elements to create something very interesting! The steps taken to create the dataset seem non-trivial and the scale of the dataset is notable (>30K functions). There is preliminary evidence that the task is non-trivial, and the authors also have interesting analysis on factors that lead to more difficult program understanding on this task. I think there is potential for the authors to leverage their ingenuity in constructing this dataset for interesting applications. After skimming CruxEval which seems to propose a similar approach, my judgment is that the underlying dataset scale and difficulty of EXE is more noteworthy.

weaknesses:
I think the motivation of this paper is great, and the creativity to create an executable programming benchmark is excellent! I think there is great potential in this work! I would recommend the authors try to focus on some of the following facets. 

1. Clarification of Test case generation methodology

I may have missed it, but I tried to look for details on methodology of test case input generation. The authors are clear on the accepted types are allowed for inputs/outpus, but it is unclear how generation is done. The best I could find is: "Based on the type definition (used for setting the function calling schema) inputs/ output pairs have been generated with the goal of maximising diversity of control flow paths within the function." and "Using the argument type annotations we construct a LLM function calling schema that generates a diverse set of inputs." The paper requires more details and clarification on this, and depending on the methodology chosen, this could affect the merits of the approach. 

2. Experiments / Lack of Models Considered

Because this is a datasets and benchmarks paper and the paper's motivation emphasizes "difficulty" of the task, not enough is done to substantiate this claim. My expectation for a dataset/benchmark paper should be at least to evaluate numerous open source models (e.g. CodeLLama, LLama3 family, CodeT5, etc) of varying sizes in addition to commercial models. Additionally, only 2 commercial models from OpenAI are used. Performing wider evaluation will strengthen these claims and the analysis, otherwise, it is an open question on how other models would perform on this task. 

3. The framing of experiments + context of other works (a potential lack of novelty)

The authors do not distinguish their approach or experiments from a dataset like CodeNet. The code understanding experiments provided here can also be done with CodeNet. If the authors could show that LLM performance or the nature of LLM performance is different on their task vs. CodeNet, this would substantiate the contribution. Of course the code on github is more diverse in nature, but on the other hand, the input/output types are still limited, and a dataset like CodeNet is multi-lingual. 

My recommendation would be to consider other creative uses of this dataset besides the ones you currently have. 


4. Polished Writing

A paper for this venue should have a higher standard of polishing. For example, the term AST should be introduced as an Abstract Syntax Tree (AST) and referred to as AST. At one point the authors colloquially refer to evaluation benchmarks as "evals." These are minor points and easy to fix, but are nevertheless are standards. 

5. Clarification on Licensing, Copyright, etc. 

I did not see clarification if the authors filtered code for permissively licensed software and if the dataset falls under acceptable use of the software.

questions:
1. Can you provide more detail on the methodology of test case input generation? Even the code used for this will work, although an explanation would help as well. 

2. Can you provide more explicit clarification on your proposed contributions, especially in context of a dataset like CodeNet. Besides the fact that the functions are from Github and that the dataset in theory can evolve, is there anything else I have misunderstood? 

3. [Recommendation to address limitation] Are you able to provide more comprehensive evaluations of other model? If the authors have access to computing resources, I strongly recommend open access models like CodeLlama to avoid API costs. If the authors have access to API credits, I would recommend at least one very large commercial model such as sonnet 3.5 or Llama3.1 405 Instruct (e.g. hosted on AWS bedrock). Although alone, I do not think these will convince me the paper should be accepted. 

4. Licensing / Copyright: Can you explain what licenses exist for the data mined for the benchmark? e.g. was filtering done for permissive licenses? Additionally if more context can be provided then if the dataset is a fair and acceptable use of the software under consideration. 

5.Clarification on Side-Effects, Determinism, and Execution Environment: Can you explain how you implement ensuring that there are "no side-effects" and that determinism indeed holds? I understand there are some banned imports, but can you provide more clarification? How do we know that this is indeed comprehensive enough to make these claims? Additionally, can you specify the python version / environment used for executing the python code? In a perfect world, it would be good to have a docker container with the same environment used to execute these programs so that the input/output examples are indeed reproducible.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730485188485

thread_type:
official_review

<thread_object>
creator: reviewer_bgbv

<content>
summary:
The authors present a new benchmark to evaluate LLMs' capability in executing real-world code. To collect a set of executable code from the real world, they built a pipeline to collect repos from GitHub to construct self-contained, deterministic code. They performed static analysis to inline the dependencies to make it self-contained, and then generated inputs using LLMs. The benchmark includes 30,000 tasks across 1,000 popular Python repos. They evaluated GPT-4o and GPT-4o mini and showed that these strong models still struggle with more complex tasks.

strengths:
* The benchmark addresses the issue in the prior work, i.e. CruxEval, by collecting real-world Python functions, instead of synthetically generated ones from LLMs.
* The benchmark includes diverse tasks and spans across 1000 repos
* The pipeline is mostly automatic and can be updated to include newer repos to address the benchmark contamination problem
* They provide analysis regarding the relationship between performance and line count, number of function calls, execution time, etc. to better understand what affects performance

weaknesses:
* The main issue with the work is that it lacks certain insights as to how this benchmark would shed light. For example, many people use CruxEval because it correlates well with model's code generation/understanding ability. Does evaluating on this benchmark instead of CruxEval serve as a better predictor of such capability?
* The paper evaluates on two models: GPT4o and GPT4o-mini. It would be better to also evaluate some open source models to compare against the closed API-only ones, especially the StarCoder model which explicitly provides training data, so one can check whether the code in the training data affects the execution prediction or not
* The input test cases are LLM generated. Since the work emphasizes real-world scenarios, it would be good to assess whether the LLM-generated test cases are of reasonable quality, and whether it gives an advantage to the LLM that generated the test cases in performing the task

questions:
* What is the model used to generate inputs? Does it matter if different models are used for input generation?
* The inlining to create a doable Python program, although necessary to make the task self-contained, also seems to make the code not look like real-world cases. Is there a way to address this?
* Are there any observations on what types of packages the LLM struggles with? Is there more we can learn if there is more thorough error analysis?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703930168

thread_type:
official_review

<thread_object>
creator: reviewer_eide

<content>
summary:
The paper introduces EXE, a new benchmark designed to evaluate language models (LLMs) on their ability to execute Python code sourced from real-world applications. This benchmark aims to address several limitations of existing evaluations, particularly the issues of scalability, task diversity, training data contamination, and benchmarking costs.  

The benchmark comprises over 30,000 tasks drawn from 1,000 popular GitHub repositories, spanning different complexities and computational operations like logical inference, mathematical reasoning, and state management. 

To construct this benchmark, the authors first select the top 1,000 most popular pypi packages and collate the corresponding github repos, after that, the authores perform a static ast analysis to filter to functions with LLM generatable argument and return type annotations. Finally, the authors apply LLM to generate test cases.

The evaluation with GPT-4 model demonstrate the limitation of existing code models.

strengths:
### 1. This paper is well-written and easy to follow.

### 2. Benchmarking code LLM is an important problem.

### 3.  The findings are interesting.

weaknesses:
## 1. The motivation for this work is not clearly articulated. 

The paper proposes benchmarking the code execution capabilities of LLMs, but it is unclear why such a capability is needed given the existing roles of compilers and interpreters. A possible motivation might be that LLMs are more lightweight and could predict execution outcomes without running the code. However, I did not see any evaluation results to support this assumption.

## 2. The paper suggests that the proposed dataset can guard against data contamination [1, 2], but lacks a detailed explanation of how this is achieved. 

The authors claim that the dataset is dynamically collected from GitHub, which could help mitigate contamination. However, since the benchmark is built from popular GitHub repositories that do not frequently change, the dataset may not be as dynamic as implied. Additionally, because the test inputs are generated by LLMs, it is unclear how this setup effectively prevents data contamination.

## 3. Certain methodological details are missing. 

First, in "Function Selection and Dependency Collation," the authors mention using static AST analysis, but it is not clear how this process is performed. Second, regarding the error metric, the authors state that they "compare the type and message (excluding stacktrace) using a language model comparison," which is described too vaguely to understand how this metric is actually computed.

## 4. This work lacks soundness in the following areas: 

(1) The authors claim the benchmark is diverse; however, there is no diversity evaluation regarding the prompts and solutions. (2) Since all test cases are generated by an LLM, there is no guarantee that the test cases are sound or appropriate for the programs. Given that some test cases result in errors during execution, this raises soundness concerns.

## 5. Minor: Some figures are of low resolution and unclear.


[1] GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

[2] PPM: Automated Generation of Diverse Programming Problems for Benchmarking Code Generation Models

questions:
1. Why we need to benchmark LLM's executation capability.

2. Can you introduce more details of the approach and the evaluation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733158053786

thread_type:
official_comment

<thread_object>
creator: reviewer_vhj6

<content>
comment:
I believe I have not heard back individually from the authors, and the review period is coming to a close. It would be good to hear back and have a response from the authors' given there is not much time remaining for review and discourse.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Reject by reviewer consensus.

The paper introduces EXE, a benchmark designed to evaluate language models (LLMs) on their ability to execute Python code sourced from github and with LLM generated test cases. Reviewers generally understood this work and liked the motivation as well as how it represents real-world Python code.

Several reviewers think that only evaluating on GPT-* models is insufficient and urged inclusion of at least some open models. Reviewers also pointed out various issues such as diversity (only Python), and whether model generated test cases are trustworthy.