Venue: NAACL 2025

CVE-Bench: Benchmarking LLM-based Software Engineering Agent's Ability to Repair Real-World CVE Vulnerabilities

Peiran Wang1, Xiaogeng Liu2, Chaowei Xiao2,

1Tsinghua University, 2University of Wisconsin-Madison,

###### Abstract

Automated vulnerability repair is a crucial field within software engineering and security research. Large Language Models (LLMs) and LLM agents have demonstrated significant potential in this domain by understanding descriptions in natural language and generating corresponding formal code. Although the coding capabilities of LLMs have advanced rapidly, evaluation benchmarks for real-world programming setups are still lagging, preventing the development of LLM and LLM agents in real-world vulnerability repair. To this end, we introduce CVE-Bench, an evaluation framework consisting of 509 Common Vulnerabilities and Exposures (CVEs) from four programming languages and 120 popular open-source repositories. Unlike previous vulnerability repair benchmarks, which only involve the code input and output, we provide LLM agents with a test environment that simulates the real-world vulnerability repair process. This environment provides multiple levels of CVE information modeling, such as black-box testing and white-box testing. It enables the agents to use static analysis tools to assist their repair process. Our evaluation reveals that the SWE-agent can only repair 21% of vulnerabilities at its best. Furthermore, they lack expert knowledge about how to use the analysis tool to assist in vulnerability repair.

## 1 Introduction

Repairing vulnerability is an essential task due to the possible significant loss brought by the vulnerability. For instance, the outbreaks of WannaCry [3] and NotPetya [4] caused a 180 billion loss in 2017, and the Equifax [6] data breach exposed the sensitive data of more than 143 million consumers. The computer security community follows a "report and repair" workflow to deal with the vulnerability [12]. Experts can report the existence of vulnerabilities by black-box or white-box penetration testing [13]. The black-box testers can only access the compiled or deployed program, so they can only report vague information on how to exploit the vulnerability. Meanwhile, the white-box testers can report rich information about where to fix the problem because they can access the source code during the mining. After that, the security expert or developers can manually repair the vulnerability based on the reported information. However, repairing software vulnerabilities based on the above information often requires significant input from human experts, who must navigate complex code bases and understand intricate inter-dependencies within the software. This process is inefficient and labor-intensive, especially when dealing with ambiguous or sparse information from sources like black-box penetration testing. As a result, the task consumes substantial human resources, highlighting the motivation to design an automated solution.

On the other hand, Large language models (LLMs) agents have been used for software development [2, 14, 15, 16, 17, 18] to reduce extensive the developer labor-intensive. To speed up the development of LLM agents for software development, diverse benchmarks have been proposed on benchmarking general code generation ability for software developments [2, 15, 16, 17], leaving the benchmarking for vulnerability vastly less explored. Despite there are some benchmarks [15, 16, 17, 18]. for vulnerability fixing, these benchmarks mainly support the code blocks as the input [15] or provide a general bug-repairing evaluation [15], without insight into the inherent rationale of the LLM-based agent execution process (e.g., interactive features with code 

interpreter, tool-using features).

In this paper, we introduce CVE-Bench (SS2), a benchmark that evaluates LLM-based agents in a realistic vulnerability-repairing setting. CVE-Bench contains three unique characteristics:

(1) Instead of input-output evaluation, CVE-Bench supports agent-based evaluation by offering real-world interactive execution-guided programming environments. Such an environment can provide feedback to the agent using the interpreter to fix vulnerabilities.

(2) CVE-Bench supports tool-using. In real-world vulnerability repair settings, developers can use diverse static or dynamic analysis tools to assist the vulnerability repair process. For instance, given vague information about a vulnerability's exploit process, developers can use the analysis tool to assist the repair process (e.g., scan and locate the vulnerability code block position and debug the repaired code).

(3) CVE-Bench simulates the vulnerability-repairing process in the real world by providing multiple-level information consideration that models white-box and black-box CVE reports. For real-world vulnerability fixing, different levels of information may provided by either the white-box or black-box analyzers (as shown in Figure 1). White-box analyzers can access the source code, locate the problematic code blocks, and report them to the developers. Thus, the developers can directly locate the vulnerability code and repair it. However, black-box analyzers can only access the compiled or applied programs and exploit the vulnerability but do not know the actual location of the code blocks. Thus, developers can only fix the vulnerability with just simple instructions.

Specifically, CVE-Bench provides 509 CVE samples across four programming languages (Python, Java, JavaScript, and PHP). The selected CVEs come from 120 open-source GitHub repositories, with at least 5000 stars for each repository. Over 16 types of CWEs are included in the collected CVEs. You can see more details about the selected CVEs in SSA. Furthermore, we provide four static analysis tools (Prospector, Bandit, Pylind, and Mypy) in our interactive environment to assist the repair process.

We evaluate our benchmark with SWE-agent Yang et al. (2024). We find that the SWE agent can achieve the repair with a success rate of about 21% at its best. Furthermore, we observe that the tested agent can only solve 14% of tasks under the black-box-level information setting. Further observation also uncovers that the agents lack the knowledge to use the analysis tool. This underscores the need to build more interactive LLM-based agents and enhance the agents' ability to use tools.

## 2 CVE-Bench

In this section, we discuss the overall design of the CVE-Bench. We first discussed constructing the

Figure 1: An example of black-box repairing and white-box repairing for LLM-based agents. We use the agent’s CVE-2019-25066 as an example: (1) Black-box hackers can only access the compiled application and realize the exploitation chain. His reported information about the repository’s vulnerability issues may only include how to exploit it but not where to repair it. (2) White-box hackers can access the source code. Thus, their reported information will include the direct script path and the corresponding code method. Thus, the LLM agents may face multiple levels of information when required to repair the vulnerability issues.

 

whole benchmark in SS2.1. Then, we formulate the task for the CVE-Bench in SS2.4. At last, we list the features of CVE-Bench that make it different from previous works and contribute to future research in SS2.3.

### Benchmark Construction

The construction process of CVE-Bench follows the three steps (see Figure 2):

**Step I: Environment construction**. CVE-Bench starts by selecting target CVEs for the benchmark. We explore the CVEFixes database [9] and select 509 CVEs from 4 programming languages, with the corresponding repositories' star counts over 5,000 (see details in SSA). Then, we query three types of information, including the CVE description, the CVE script, and the CVE method/function, and use the three types of information to construct three levels of information to model the black-box (description), white-box (description + script + method) reports, and an intermediate level between the two (description + script). After the information is constructed, CVE-Bench builds the code base for the evaluation. CVE-Bench first clones the CVEs' corresponding repositories, then queries the database for the CVEs' repair belonging commits and checkouts the repository to the parent commits.

**Step II: Repair patch generation**. Next, CVE-Bench sends the generated information to the agent. The agent builds a docker environment for the constructed code base when receiving the input information. Then, the agent performs an iterative repair process: (1) The LLM generates the instruction to be executed; (2) The agent inputs the instruction on the terminal using the docker; (3) The terminal execute and output feedback; (4) The feedback is sent back to the LLM to repeat (1)-(4). During the patch generation process, CVE-Bench enhances the executable environment by providing four static analysis tools (Prospector, Pylint, Bandit, and Mypy) for the agents to call to simulate the real-world vulnerability repair environment (see detail in SSC).

**Step III: Repair validation**. At last, CVE-Bench performs execution-based repair patch validation using the unit tests. We crafted unit tests (including the Dockerfile to construct the environment for the unit test and the unit test scripts) for each selected CVE. The unit test is written following the exploit chain of each CVE. When the repair patch is generated, CVE-Bench will apply the patch to the code base and execute the unit test to judge whether the CVE is repaired.

### Task Formulation

**Model input**. First of all, CVE-Bench extracts three types of information from the CVEFixes database [8]: (1) Description: This information describes the rationale and the exploit logic of the CVE, described by natural language; (2) Script: This describes which script contains vulnerability issues, along with the script's path from the code base; (3) Method: This information detailedly describes which method triggers the vulnerability issues.

Based on these three types of information, CVE-Bench further constructs three levels of information to model real-world vulnerability report information: (1) Black box: This level only contains the description information, modeling black-box vulnerability report; (2) White box: This level contains all the description, script, and method information to model the white-box vulnerability report. See a detailed example in Figure 2. Additionally, we also provide the Intermediate category: This level contains description and script information, modeling an intermediate state between the black box and white box;

**Evaluation metrics**. We apply the generated patch to the code base using GitHub's patch program to validate the generation and execute the unit tests associated with the target vulnerabilities. If the patch applies successfully and the unit test passes, we consider the generated patch to have successfully repaired the vulnerability. The metric for our benchmark is the percentage of resolved vulnerabilities (repair rate).

### Features of CVE-Bench

Traditional benchmarks in NLP typically involve only short input and output sequences and consider somewhat "contrived" problems explicitly created for the benchmark. In contrast, CVE-Bench 's realistic construction setting imbues the dataset with unique properties, which we discuss below.

**Real-world CVE vulnerability repair tasks**. Since each task instance in CVE-Bench consists of a large and comply CVE-Bench requires demonstrating sophisticated skills and knowledge possessed by experienced software engineers but are 

not commonly evaluated in traditional code generation benchmarks.

**Real-world CVE vulnerability repair process modeling**. Unlike previous benchmarks, which only provide one level of information to evaluate LLMs, \(\mathsf{CVE}\)-Bench provides multiple levels of information as input. This approach analyzes what the LLM-based agent needs to repair a vulnerability issue. By doing so, we can dive deep into an observation about enhancing the agents' ability. This approach also aligns with the real-world vulnerability repair settings (low information input-black box, rich information input-white box).

**Interactive repair environment simulation**. Another feature of the LLM-based agent is its ability to use tools. Previous research only evaluates an agent's ability to move from input to output, treating agents as black boxes without concern and observing their inside functionality. Unlike them, \(\mathsf{CVE}\)-Bench encompasses the analysis tool using the vulnerability issues repair approach. We evaluate whether the agent chooses and uses the tool effectively.

**Execution-based validation**. Unlike previous vulnerability repairing benchmarks, which only use the sequence accuracy as the repair metrics, \(\mathsf{CVE}\)-Bench uses the unit test pass portion as the repair portion of the CVEs. This is a more reliable and robust evaluation metric since the LLMs may generate diverse but equally effective repair patches.

## 3 Evaluation

We calculated the unit test pass rate as the patch repair rate for the evaluation metrics. Furthermore, we also categorized the failed reasons as follows:

* Locate fail: LLM can not locate the vulnerability codes.
* Loop out: The repairing process reaches the maximum step limit.
* Unit test fail: The generated repair patch does not pass the unit test.
* Others: Other reasons for the agents' generation failure (e.g., failure to generate formal command).

### Different Level Information

**Method**. We compare the SWE-agent's ability to repair the vulnerability across three information levels (Black-box, intermediate, and white-box). We evaluate the agents on the selected Python CVEs, with the GPT-4 as the foundation model backend. We summarized the patch repair rate using the validation and counted the reasons for failure. We also counted the fixed rate's CDF as the interaction step increased.

**Results**. As shown in Figure 3 (a), we can observe that the repair rate under the white-box setting is better than the black-box setting. When under the black-box setting, the repair rate is only

Figure 2: There are four steps in \(\mathsf{CVE}\)-Bench: (1) Environment construction: \(\mathsf{CVE}\)-Bench uses the three-level information to generate a vulnerability issue as the input to the agent. \(\mathsf{CVE}\)-Bench clones the CVEs’ corresponding repositories and checks out the repository to the parent commits. (2) Repair patch generation: Next, \(\mathsf{CVE}\)-Bench sends the queried information divided into multiple levels to the agents for patch generation. (3) Repair validation: At last, \(\mathsf{CVE}\)-Bench performs execution-based repair patch validation using the unit tests. \(\mathsf{CVE}\)-Bench also considers comparing the generated patches with the ground truth repair code crafted by the man.

 

about 14.1%. While under the intermediate setting, the repair rate reaches about 17.8%. The white-box setting gives the agent the best ability to repair the vulnerability issues, with a repair rate of over 21%.

Then, we studied the failure reasons across the three levels of information in Figure 3 (a). Under the black-box setting, the most common reason for failure commands is that the agent can not locate the vulnerability. This aligns with our intuition: the agent cannot independently locate the detailed location (script and method). The same observation also occurs in the intermediate setting, indicating that the agent can not locate where the vulnerability is even when the script path is given. We checked the data for the white-box setting and found that the main reason for failure was failing to pass the unit test. This indicates that the white box can always craft the repair patch but not a stable one.

For the success repair, we can observe more details in Figure 3 (b). For the white-box setting, the agent can quickly fix the vulnerability within low steps because it knows the actual position of the code blocks. The other two levels (black box and intermediate) require more steps to accomplish the tasks because the agent needs to find the location of the vulnerability code blocks in the first few steps.

### Programming Languages

**Method**. We compare the performance of the SWE-agent when solving vulnerability repair tasks for different programming languages (Python, Java, JavaScript, and PHP). For this test, we only consider the GPT-4 foundation model backend and the information at the white-box level as input.

**Results**. The results are shown in Figure 4. The SWE-agent performs best when repairing the Python CVEs with the highest repair rate of 21.17%. The repair rates on JavaScript (17.81%) and PHP (16.33%) are nearly the same and fall about 5% behind the repair rate on Python. However, the SWE-agent performs worst when solving Java CVEs with a repair rate of around 12%. This may be due to the GPT-4's training dataset, which may contain more knowledge of Python than Java.

\begin{table}
\begin{tabular}{l l l l l} \hline Info. & Python & Java & JavaScript PHP \\ Level & & & & \\ \hline Repair \# & 15 & 53 & 127 & 38 \\ Repair \% & 21.17 & 12.33 & 17.81 & 16.33 \\ Edit line & 74.00 & 34.90 & 43.23 & 34.34 \\ Edit word & 3206.00 & 1735.98 & 1606.73 & 2010.42 \\ \hline \end{tabular}
\end{table}
Table 1: We evaluate the tested agents across four programming languages: 1)Python, 2)Java, 3)JavaScript, and 4)PHP.

Figure 4: We evaluate the tested agents across four programming languages: 1)Python, 2)Java, 3)JavaScript, and 4)PHP.

Figure 3: We evaluated the tested agents using three different levels of information: CVE description, CVE script, and CVE method to model the real-world vulnerability fixing (black box and white box). (a) We summarized the repair rate and the three reasons for failure at each level. (b) We counted the CDF of the fixed rate among the three levels of information with the interaction step increase.

 

Furthermore, the syntax complexity of Java is also higher than the complexity of the other three languages.

### Different Foundation Models: GPT-4 vs. GPT-3.5 vs. Llama-3

**Method**. We compare the agent's ability to repair the vulnerability under the GPT-4, GPT-3.5, and the Llama models. Due to the high cost, we set the information level to the white-box level, with 50 Python CVE samples as the test set.

**Results**. Figure 5(a) lists the patch repair portion. We can observe that the repair portion of GPT-4 is the highest. While using GPT-3.5 and Llama, the repaired portion drops quickly.

Then, we analyzed the failure reasons for the three different foundation models. As shown in Figure 5(a), for GPT-4, the most common failure reason for failed commands is unit test failure. The most common reason for failure in GPT-3.5 is the failure to generate formal commands (including in others). This is because GPT-3.5 can not generate formal commands to execute on the terminal. It always falls short of generating unuseful or syntax error commands.

### Tool Using

**Method**. We compare the performance of the SWEGRET when using different static analysis tools and when not using tools. We only considered Python CVEs and used 50 testing samples to simplify the experiment. The information level is set to the black-box setting. We consider four distinct tools: Pylint, Bandit, Mypy, and Prospector (see details in SSC). We compare the repair rate and the repair CDF changes across the four tools used.

**Results**. As shown in Figure 6 (a), the tool used does not enormously increase the repair portion; some tools even make the repair rate drop compared with non-tool use (Default). This may be attributed to the fact that the LLM does not have enough knowledge to use these tools. Also, using the tool requires more loops of interactions, which may cause the agents to reach the maximum limit of the loops.

Furthermore, we observe in Figure 6(b) the tools require the agents to take more steps to repair a vulnerability. However, we believe the agent failed to use the tool effectively because the repair rate did not increase greatly.

### Evaluation for Overfitting

**Method**. Considering the potential overfitting issues for the GPT-4 backend for the tested agents, we compare the repair rate between the repair cases on CVE-Bench's pre-collected CVEs and newly collected CVEs after 2023 Nov (the end of GPT-4 training). We collected 50 Python CVEs after 2023 Nov to avoid the potential training sample involved. Then, we used the CVE-Bench under GPT-4 back-end, with the information level set to white-box setting, to test the agents on the 50 Python CVEs.

**Results**. As shown in Figure 7 (a), the repair rate on recent CVEs is lower than that of pre-collected CVEs. This indicated that the pre-trained samples strongly affected the agents' repair ability. If the related knowledge persists in the agents' backend foundation models' training samples, it may be easier for them to locate and repair the vulnerability issues.

The CDF in Figure 7 (b) further reveals the overfitting effect. For the pre-collected CVE, the agent mainly takes less than 12 steps to accomplish the repair tasks. Meanwhile, for the recent CVE, the agent must take more steps to accomplish the tasks.

Figure 5: We evaluate the tested agents across three different foundation models: 1)GPT-4, 2)GPT-3.5 and 3)Llama-3.

 



### A quantitative analysis

We provide an example from CVE-2022-0577, a Python CVE from the Scrapy repository (see Figure 8). The CVE's description is "Exposure of Sensitive Information to an Unauthorized Actor in GitHub repository scrapy/scrapy."

In this example, the agents are given the 3-level information contained in the generated issue. The agent adds four lines of code in the agent-generated code blocks to check whether the header is from the two sensitive headers: "Cookie" and "Authorization," to repair the potential vulnerability. However, as we observe in Figure 8's ground truth repair code, the developers added another function to check the header detailedly. Furthermore, the developers call another function from the other script to solve it within the created function.

This highlights current LLM-based agents' drawbacks: First, it tends to repair the code directly in its original location. Compared to code written by human developers, it is not concise enough. It lacks maintainability (adding a lot of code in place is more likely to make code development more confusing than creating a specific function). Second, the LLM agents still lack enough domain to call the existing function to solve the problem. Still, they aim to fix the problem using their own generated code, which is inconsistent with the modular development characteristics of software engineering.

## 4 Related Work

**Code generation benchmark**. HumanEval Chen et al. (2021) has become the benchmark for the enduring challenge of generating code from natural language specifications Yu et al. (2018); Austin et al. (2021); Hendrycks et al. (2021a, b); Zan et al. (2023). Over the past year, various benchmarks have aimed to enhance HumanEval by incorporating support for multiple programming languages Cassano et al. (2022); Athiwaratkun et al. (2023); Orlanski et al. (2023), diverse editing scopes Yu et al. (2023); Du et al. (2023). Innovative code completion challenges Muennighoff et al. (2023), along with an increased emphasis on testing Liu et al. (2023b, a). In parallel, other studies have endeavored to introduce innovative coding methodologies Yin

Figure 6: We evaluate the tested agents across four different static analysis tool support: Pylint, Bandit, mypy, and Prospector. We compare the repair rate and the CDF of repair between steps.

Figure 7: We compare the repair rate between the repair cases on CVE–Bench’s pre-collected CVEs and newly collected CVEs after 2023 Nov (the end of GPT-4 training) to avoid the effect of overfitting.



et al., 2022; Yang et al., 2023) or to develop problems tailored to specific libraries Lai et al. (2022); Zan et al. (2022). Instead of general code generation problems, CVE-Bench focuses on vulnerability repair problems. Vulnerability repair requires the agents to understand the vulnerability exploit rationale from the attackers' view. Based on such understanding, the agents can generate high-quality repair patches. Furthermore, real-world vulnerability repair involves black-box testing reports, which only provide vague information for the agents.

**ML for software engineering**. Machine Learning (ML) is revolutionizing Software Engineering by automating software development processes using neural networks and Large Models (LMs) Zheng et al. (2023); Hou et al. (2023). This includes automating commit messages Jung (2021); Liu et al. (2023a), enhancing PR reviews Yang et al. (2016); Li et al. (2022b), bug localization Kim et al. (2019); Chakraborty et al. (2018), software testing Kang et al. (2023); Xia et al. (2023), and program repair Gupta et al. (2017); Allamanis et al. (2017). For CVE-Bench, relevant works focus on applying LMs to automated program repair Xia and Zhang (2022); Fan et al. (2023) and guiding code editing through commits Chakraborty et al. (2018); Zhang et al. (2022). However, the previous fixing mainly follows the one-time input-to-output paradigm and ignores the interactive nature of the LLM-based agents. CVE-Bench simulates a real-world environment for the program repair by providing real-world-level information and tool use.

## 5 Conclusion

Automated vulnerability repair is a popular and valuable software engineering and security research field. Large language models (LLMs) and LLVM agents have shown sizeable potential application value in this area. LLMs can understand natural language described vulnerability rationale and generate formal code to repair it. However, evaluation benchmarks modeling real-world vulnerability repair environments are still lacking. To this end, we introduce CVE-Bench, an evaluation framework consisting of 509 CVEs from four programming languages and 120 popular open-source repositories. Unlike previous vulnerability repair benchmarks, which only involve the code input and output, we provide LLM agents with a real-world vulnerability repair environment. In this environment, multiple levels of CVE information modeling, such as black-box and white-box testing, are provided to rate agents' ability to repair vulnerabilities. Furthermore, we enable the agents to use static analysis tools in this environment to assist their repair process and see how the repair rate changes with the tool's use. Our evaluation shows that the agent's ability differs greatly from white to black-box settings. And the agent can still not use the static analysis tool effectively.

Figure 8: We show an example of a formatted task instance, the original code, ground-truth repair code (extracted from the database, made by the developers), the 3-level information, generated issue, and the agent-generated code. In the code blocks, grey highlights are additions.

 



## 6 Limitation

**Limited programming language coverage**. Because there are too many programming languages, our benchmark cannot cover too many types. At the same time, due to the high complexity of some programming languages, it is difficult for us to build a more robust unit test set.

**Limited tested agent types**. Since there are relatively few existing agent types, we only conducted experiments on SWE-Agent Yang et al. (2024). Furthermore, Open-Devin OpenDevin Contributors (2024) does not provide a sufficient API for us to call. It only supports using its API, which limits CVE-Bench to establish large-scale experiments.

**Limited tested foundation models**. Until our paper is submitted, we have only performed the experiments on the GPT -4, GPT-3.5, and Llama-3 models. We plan to add more experiments before the publication.





## References

* A. Aidan, H. Verma, and L. K. Awasthi (2017)Comprehensive survey on petya ransomware attack. In 2017 International conference on next generation computing and information systems (ICNGCIS), pp. 122-125. Cited by: SS3.4.
* M. Allamanis, M. Brockschmidt, and M. Khademi (2017)Learning to represent programs with graphs. arXiv preprint arXiv:1711.00740. Cited by: SS1.
* B. Athiwaratkun, S. Gouda, Z. Wang, X. Li, and Y. Tian et. al. (2023)Multi-lingual evaluation of code generation models. PreprintarXiv:2210.14868. External Links: Link Cited by: SS1.
* J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Johan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. (2021)Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Cited by: SS1.
* G. Bhandari, A. Naseer, and L. Moonen (2021)CVEfixes: automated collection of vulnerabilities and their fixes from open-source software. In Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering (PROMISE '21), New York, NY, USA, pp. 10. Cited by: SS1.
* Q. Bui, R. Scandariato, and N. E. Trizareyra (2022)Vul4j: a dataset of reproducible java vulnerabilities geared towards the study of program repair techniques. In Proceedings of the 19th International Conference on Mining Software Repositories, MSR '22, New York, NY, USA, pp. 464-468. External Links: ISBN 978-1-4673-319-1, Document, Link Cited by: SS1.
* F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M. Yee, Y. Zi, C. Jane Anderson, M. Q. Feldman, A. Guha, M. Greenberg, and A. Jangda (2022)Multipl-e: a scalable and extensible approach to benchmarking neural code generation. PreprintarXiv:2208.08227. Cited by: SS1.
* USERT (2017)Indicators associated with wannacry cansonware. External Links: Link Cited by: SS1.
* S. Chakraborty, Y. Li, M. Irvine, R. Saha, and B. Ray (2018)Entropy guided spectrum based bug localization using statistical language model. arXiv preprint arXiv:1802.06947. Cited by: SS1.
* M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Ponde de Oliveira Pinto, and J. Kaplan et. al. (2021)Evaluating large language models trained on code. PreprintarXiv:2107.03374. Cited by: SS1.
* X. Du, M. Liu, K. Wang, H. Wang, J. Liu, Y. Chen, J. Feng, C. Sha, X. Peng, and Y. Lou (2023)Classeval: a manually-crafted benchmark for evaluating lms on class-level code generation. PreprintarXiv:2308.01861. Cited by: SS1.
* Z. Fan, X. Gao, M. Mirchev, A. Roychoudhury, and S. H. Tan (2023)Automated repair of programs from large language models. PreprintarXiv:2205.10583. Cited by: SS1.
* R. Gupta, S. Pal, A. Kanade, and S. Shevade (2017)DeepFix: fixing common c language errors by deep learning. In Proceedings of the aaai conference on artificial intelligence, Vol. 31. Cited by: SS1.
* D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, Akul Arora, Ethan Guo, Collin Burns, S. Puraniak, H. He, D. Song, and J. Steinhardt (2021)Measuring coding challenge competence with apps. PreprintarXiv:2105.09938. Cited by: SS1.
* D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puraniak, H. He, D. Song, and J. Steinhardt (2021)Measuring coding challenge competence with apps. NeurIPS. Cited by: SS1.
* X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang (2023)Large language models for software engineering: a systematic literature review. PreprintarXiv:2308.10620. Cited by: SS1.
* N. Jain, S. Vaidyanath, A. Iyer, N. Natarajan, S. Parthasarathy, S. Rajamani, and R. Sharma (2022)Jigsaw: large language models meet program synthesis. In Proceedings of the 44th International Conference on Software Engineering, pp. 1219-1231. Cited by: SS1.
* Z. Ji, P. Ma, Z. Li, and S. Wang (2023)Benchmarking and explaining large language model-based code generation: a causality-centric approach. arXiv preprint arXiv:2310.06680. Cited by: SS1.
* C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan (2023)Swe-bench: can language models resolve real-world github issues?. arXiv preprint arXiv:2310.06770. Cited by: SS1.
* T. Jung (2021)Commitbert: commit message generation using pre-trained programming language model. PreprintarXiv:2105.14242. Cited by: SS1.
* S. Kang, J. Yoon, and S. Yoo (2023)Large language models are few-shot testers: exploring llm-based general bug reproduction. PreprintarXiv:2209.11515. Cited by: SS1.
* Yunho Kim, Seokhyeon Mun, S. Yoo, and M. Kim (2019)Precise learn-to-rank fault localization using dynamic and static features of target programs. ACM Transactions on Software Engineering and Methodology (TOSEM)28 (4), pp. 1-34. Cited by: SS1.
* Yuhang Lai, Chengxi Li, Y. Wang, T. Zhang, R. Zhong, L. Zettlemoyer, S. Wentau Yih, D. Fried, S. Wang, and T. Yu (2022)Ds-1000: a natural and reliable benchmark for data science code generation. PreprintarXiv:2211.11501.



* Li et al. (2022a) Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Remi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022a. Competition-level code generation with alphacode. _Science_, 378(6624):1092-1097.
* Li et al. (2022b) Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sundaresan. 2022b. Automating code review activities by large-scale pre-training. _Preprint_, arXiv:2203.09095.
* Liu et al. (2024) Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. _Advances in Neural Information Processing Systems_, 36.
* Liu et al. (2023a) Shangqing Liu, Yanzhou Li, Xiaofei Xie, and Yang Liu. 2023a. Commitbart: A large pre-trained model for github commits. _Preprint_, arXiv:2208.08100.
* Liu et al. (2023b) Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, and Hanyu Lai et. al. 2023b. Agentbench: Evaluating llms as agents. _Preprint_, arXiv:2308.03688.
* Mu et al. (2018) Dongliang Mu, Alejandro Cuevas, Limin Yang, Hang Hu, Xinyu Xing, Bing Mao, and Gang Wang. 2018. Understanding the reproducibility of crowd-reported security vulnerabilities. In _27th USENIX Security Symposium (USENIX Security 18)_, pages 919-936.
* Muennighoff et al. (2023) Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. 2023. Octopack: Instruction tuning code large language models. _Preprint_, arXiv:2308.07124.
* Nijkamp et al. (2022) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Liftu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. _arXiv preprint arXiv:2203.13474_.
* Contributors (2024) OpenDevin Contributors. 2024. Opendevin: Code less, make more. [https://github.com/OpenDevin/OpenDevin](https://github.com/OpenDevin/OpenDevin).
* Orlanski et al. (2023) Gabriel Orlanski, Kefan Xiao, Xavier Garcia, Jeffrey Hui, Joshua Howland, Jonathan Malmaud, Jacob Austin, Rishabh Singh, and Michele Catasta. 2023. Measuring the impact of programming language distribution. _Preprint_, arXiv:2302.01973.
* Verma et al. (2017) Akanksha Verma, Amitta Khatana, and SArika Chaudhary. 2017. A comparative study of black box testing and white box testing. _International Journal of Computer Sciences and Engineering_, 5(12):301-304.
* Wang et al. (2024)Xinchen Wang, Ruida Hu, Cuiyun Gao, Xin-Cheng Wen, Yujia Chen, and Qing Liao. 2024. Reposvul: A repository-level high-quality vulnerability dataset. In _Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings_, ICSE-Companion '24, page 472-483, New York, NY, USA. Association for Computing Machinery.
* Xia et al. (2023) Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. 2023. Universal fuzzing via large language models. _arXiv preprint arXiv:2308.04748_.
* Xia and Zhang (2022) Chunqiu Steven Xia and Lingming Zhang. 2022. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In _Proceedings of the 30th ACM Joint European Software Enginering Conference and Symposium on the Foundations of Software Engineering_, pages 959-971.
* Yang et al. (2024) John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunya Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering.
* Yang et al. (2023) John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. 2023. Intercode: Standardizing and benchmarking interactive coding with execution feedback. _Preprint_, arXiv:2306.14898.
* Yang et al. (2016) Xin Yang, Raula Gaikovina Kula, Norihiro Yoshida, and Hajim Iida. 2016. Mining the modern code review repositories: A dataset of people, process and product. In _Proceedings of the 13th International Conference on Mining Software Regositories_, pages 460-463.
* Yin et al. (2022) Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yaming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryx Michalewski, Alex Polozov, and Charles Sutton. 2022. Natural language to code generation in interactive data science notebooks. _Preprint_, arXiv:2212.09248.
* Yu et al. (2023) Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Tao Xie, and Qianxiang Wang. 2023. Codereval: A benchmark of pragmatic code generation with generative pretrained models. _Preprint_, arXiv:2302.00288.
* Yu et al. (2018) Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. _arXiv preprint arXiv:1809.08887_.
* Zan et al. (2022) Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Jian-Guang Lou. 2022. Cert: Continual pre-training on sketches for library-oriented code generation. _Preprint_, arXiv:2206.06888.
* Zan et al. (2023)Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Yongji Wang, and Jian-Guang Lou. 2023. Large language models meet nl2code: A survey. _Preprint_, arXiv:2212.09420.
* Zhang et al. (2022) Jiyang Zhang, Sheena Pantahaplackel, Pengyu Nie, Junyi Jessy Li, and Milos Gligoric. 2022. Codit5: Pretraining for source code and natural language editing. _Preprint_, arXiv:2208.05446 

Zibin Zheng, Kaiwen Ning, Jiachi Chen, Yanlin Wang, Wenqing Chen, Lianghong Guo, and Weicheng Wang. 2023. Towards an understanding of large language models in software engineering tasks. _arXiv preprint arXiv:2308.11396_.
* Zou et al. (2018) Yixin Zou, Abraham H Mhaidli, Austin McCall, and Florian Schaub. 2018. " i've got nothing to lose": Consumers' risk perceptions and protective actions after the equitax data breach. In _Fourteenth Symposium on Usable Privacy and security (soups 2018)_, pages 197-216.

 

DTABLE]

## Appendix A Dataset Details

In this section, we discussed the details of the CVE dataset.

**Programming languages**. The CVE dataset contains 509 vulnerability issues selected from four programming languages (Java, Python, JavaScript, and PHP). The portion of each programming language is shown in Figure 9(a) (JavaScript: 164, PHP: 124, Python: 124, Java: 87).

**CVE severity&CVE score**. The severity of CVEs (Common Vulnerabilities and Exposures) is classified using the CVSS (Common Vulnerability Scoring System). The CVSS provides a way to capture the principal characteristics of a vulnerability and produce a numerical score reflecting its severity. The numerical score can then be translated into a qualitative representation (low, medium, high, and critical) to help organizations properly assess and prioritize their vulnerability management processes. Here are the general severity ratings based on the CVSS score:

1. None: 0.0
2. Low: 0.1 - 3.9
3. Medium: 4.0 - 6.9
4. High: 7.0 - 8.9
5. Critical: 9.0 - 10.0

The CVSS score is calculated based on several metrics, which fall into three main groups:

* **Base metrics**: These represent a vulnerability's intrinsic and fundamental characteristics that are constant over time and across user environments. Base metrics include the attack vector, attack complexity, privileges required, user interaction, scope, confidentiality, integrity, and availability impacts.
* **Temporal metrics**: These reflect the characteristics of a vulnerability that may change over time but not across user environments. Temporal metrics include the exploit code maturity, remediation level, and report confidence.
* **Environmental metrics**: These represent the characteristics of a vulnerability that are relevant and specific to a particular user's environment. Environmental metrics include collateral damage potential, target distribution, security requirements, and modified versions of the base metrics.

We summarized the collected CVEs severity distribution in Figure 9(b) and the CVSS score distribution in Figure 10.

The overall CVSS score combines these metrics to provide a standardized view of a vulnerability's severity.

**CWE type of the selected CVEs**. The Common Weakness Enumeration (CWE) is a community-developed list of common software and hardware security weaknesses. Maintained by the MITRE Corporation, CWE aims to serve as a standard reference point for identifying, mitigating, and preventing weaknesses in software and systems throughout their lifecycle. By providing a unified language for discussing software security issues, CWE helps improve the quality of software and hardware products by reducing the prevalence of vulnerabilities.

CWE uses a hierarchical classification system to categorize vulnerabilities based on their nature, cause, and impact. This classification system is

Figure 10: The distribution of the CVSS scores across the four programming languages CVEs.

Figure 9: The programming languages and the CVE severity of the selected CVEs.

 

* [25] M. C. Bader, A. D. Krizhevsky, and R. G. Hinton. Learning to predict the presence of a large number of people in an image. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1-40, 2016.
* [26] J. L. Friedman, S. W. Teh, and T. Salakhutdinov. Unsupervised learning for image recognition. In _Advances in neural information processing systems_, volume 389 of _Lecture Notes in Artificial Intelligence_, pages 77-96, 2000.
* [27] H. F. He, Y. Zhang, X. Ren, and Z. Sun. Deep residual learning for image recognition. In _IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1-10, 2016.
* [28] P. I. Goodfellow, N. Mirza, and E. Rabinovich. Generative adversarial nets. In _NIPS'05_, volume 1: MIT press, 2005.
* [29] V. Mnih, I. Sutskever, and I. Darrell. Imagenet classification with deep convolutional networks. In _Neural machine translation (ICLR)_, volume 349 of _Lecture Notes in Neural Information Processing Systems_, pages 1-10, 2005.

[MISSING_PAGE_POST]

 

the new data set, we can also be able to identify the relevant features of the dataset.

#### 4.2.3 Feature Selection and Classification

The feature selection is a key step in the classification process. The feature selection algorithm is based on the feature selection procedure described above:

* **Feature Selection and Classification**: We use the feature selection method as an input for feature selection.
* **Feature Selection**: We use the feature selection technique described below:
* **Feature Selection**: We use the feature selection method described above:
* **Feature Selection**: We use the feature selection method described above:
 

3. Our selection on the repositories follow the rule that their stars should be over at least 5000.

## Appendix B Patch Generation Details

**Database extraction**. We built a database by following [1]'s CVE collection process. This comprehensive vulnerability dataset is automatically collected and curated from Common Vulnerabilities and Exposures (CVE) records in the public U.S. National Vulnerability Database (NVD). The goal is to support data-driven security research based on source code and source code metrics related to fixes for CVEs in the NVD by providing detailed information at different interlinked levels of abstraction, such as the commit, file, and method levels, as well as the repository and CVE levels. The database structure is shown in Figure 12. We selected the CVE description (cve.description), CVE scripts (file_change.file_name), and CVE method (method_change.name) from the database to construct the three-level inputs for the CVE-Bench.

**Issue generation**. Then, we perform an issue generation to generate the natural language-described issue as the input to the tested agents.

For the CVE desc level, we only provide the CVE description queried from the database. As for the CVE script level, the ground truth fix commit's modified script filename and path will be added to the issue. Finally, the CVE method level allows us to add the ground truth fix commit's modified method name and path to the issue.

Then, the generated issue will be input to the agents for code generation. The agents will create the patch to fix the vulnerability issues.

## Appendix C Analysis Tool Details

The details of the four distinct analysis tools are as follows:

* Pylint: Pylint is a highly versatile Python static analysis tool that helps improve code quality and conform to coding standards. It checks for errors, enforces a coding standard, and looks for code smells. Pyllint offers support for customizing the rules during the analysis, allowing developers to tailor the tool to their needs. It provides detailed reports on code quality, which can be integrated into development environments and continuous integration systems.
* Bandit: Bandit is a tool specifically designed for finding common security issues in Python code. It scans Python programs to identify vulnerabilities like SQL injection, cross-site scripting, hardcoded passwords, and more. Bandit is highly configurable, allowing users to exclude certain tests or files, and can be easily integrated into CI/CD pipelines for automated security testing.
* Mypy: Mypy is a static type checker for Python. By adding type annotations to Python code (using Python's typing module), mypy checks code and detects type errors before runtime, which can significantly improve code reliability and maintainability. Myrp supports Python's dynamic typing features while adding the benefits of static type checking, making it a powerful tool for large codebases.
* Prospector: Prospector is a comprehensive linting tool that bundles multiple Python static analysis tools such as Pylint, Bandit, and Mpyy, among others. Running these tools simultaneously provides an aggregate view of code quality and potential issues. The prospector aims to simplify the setup and configuration of multiple linters and static analysis tools, providing a unified output that helps developers focus on improving code quality.

## Appendix D Limitation

**Limited programming language coverage**. Because there are too many programming languages, our benchmark cannot cover too many types. At the same time, due to the high complexity of some programming languages, it is difficult for us to build a more robust unit test set.

**Limited tested agent types**. Since there are relatively few existing agent types, we only conducted experiments on SWE-Agent [46]. Furthermore, Open-Devin [7] does not provide a sufficient API for us to call. It only supports using its API, which limits CVE-Bench to establish large-scale experiments.

**Limited tested foundation models**. Until our paper is submitted, we have only performed the experiments on the GPT -4, GPT-3.5, and Llama-3 models. We plan to add more experiments before the publication 

* [25] M. C. Bader, A. K. Goyal, and J. S. Murray, "A survey of the world-wide web for large scale data mining: a review," in _Proceedings of the 10th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 347-368, 2019.
* [26] R. D. Barnett, "The world-wide 

* [14] M. C. K. Chan, A. D. Martin, and R. J. Sivic. Learning from a single image database: A review of the state-of-the-art deep learning approach. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 2357-2360, Dec. 2018.
* [15] P. L. Barnett, G. B. Hinton, and I. V. Mnih. Deep residual networks for large scale machine translation. In _Advances in neural information processing systems_, volume 9, pages 135-140, 2016.
* [16] T. H. Cover, N. W. Tehgal, and E. Rabinovich. On the problem of overfitting. In _Proceedings of the 1st International conference on Machine Learning Research (ICML)_, pages 1-10, 2002.
* [17] Y. Bengio et al. The MIT Press, Cambridge University Press, 2005.
* [18] X. Chen, Z. Liu, and Q. Weinberger. Denschap: A survey on stochastic optimization. _arXiv preprint arXiv:1412.6097_, 2014.
* [19] Y. Bengio, O. LeCun, and J. Schmidhuber. Learning representations by jointly connected layers. In _NIPS Workshop on Neural Networks_, pages 1-10, 2015.
* [20] Y. Bengio, A. Courville, and J. Dean. Dropout: An efficient way to prevent it from overfitting. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [21] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [22] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [23] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [24] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [25] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [26] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [27] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [28] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [29] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [30] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [31] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [32] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [33] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [34] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [35] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [36] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [37] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [38] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [39] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [40] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [41] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [42] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [43] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [44] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [45] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [46] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [47] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [48] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [49] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [50] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [51] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [52] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [53] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [54] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [55] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [56] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [57] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [58] Y. Bengio, A. Courville, and J. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [59] Y. Bengio, A. Courville, and J. Dean. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [60] Y. Bengio, A. Courville, and J. Dean. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [61] Y. Bengio, A. Courville, and J. Dean. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [62] Y. Bengio, A. Courville, and J. Dean. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [63] Y. Bengio, A. Courville, and J. Dean. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [64] Y. Bengio, A. Courville, and J. Dean. Dean. Dropout: A new method for reducing the dimensionality of data. In _Proceedings of the 1st international conference on computer vision and pattern recognition_, pages 1-10, 2015.
* [65] Y. Bengio, A