<figure><img src="image_1.png" /></figure>

HAL
open science

# LLM-CVX: A Benchmarking Framework for Assessing the Offensive Potential of LLMs in Exploiting CVEs

Mohamed Amine El Yagouby, Abdelkader Lahmadi, Mehdi Zakroum, Olivier Festor, Mounir Ghogho

<figure><img src="image_2.png" /></figure>

## To cite this version:

Mohamed Amine El Yagouby, Abdelkader Lahmadi, Mehdi Zakroum, Olivier Festor, Mounir Ghogho. LLM-CVX: A Benchmarking Framework for Assessing the Offensive Potential of LLMs in Exploiting CVEs. AISec 2025 - 18 th ACM Workshop on Artificial Intelligence and Security, Association for Computing Machinery, Oct 2025, Taipei, Taiwan. 10.1145/3733799.3762978. hal-05349650

HAL Id: hal-05349650

https://inria.hal.science/hal-05349650v1

Submitted on 5 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# LLM-CVX: A Benchmarking Framework for Assessing the Offensive Potential of LLMs in Exploiting CVEs

Mohamed Amine El yagouby
CNRS, Inria, LORIA
Université de Lorraine
Nancy, France
TICLab
International University of Rabat
Rabat, Morocco
mohamed-amine.el-yagouby@loria.fr

Abdelkader Lahmadi
CNRS, Inria, LORIA
Université de Lorraine
Nancy, France
abdelkader.lahmadi@loria.fr

Mehdi Zakroum
TICLab
International University of Rabat
Rabat, Morocco
mehdi.zakroum@uir.ac.ma

Olivier Festor
CNRS, Inria, LORIA,
Université de Lorraine
Nancy, France
olivier.festor@univ-lorraine.fr

Mounir Ghogho
University Mohammed VI Polytechnic
Rabat, Morocco
Mounir.GHOGHO@um6p.ma

## Abstract

The increasing capabilities of Large Language Models (LLMs) in code generation and reasoning have raised concerns about their potential misuse, particularly for automating or assisting with vulnerability exploitation tasks. This concern highlights the need for a systematic evaluation of the offensive potential of LLMs. Existing methodologies in this context use synthetic vulnerabilities, rely on fixed prompting strategies and exploit tools in their evaluation, and do not consider efficiency. In this paper, we introduce **LLM-CVX**, a novel benchmarking framework designed to systematically evaluate LLMs on real-world CVE exploitation tasks, with extensibility towards prompting strategies and exploit tools, and allowing LLMs to make multiple attempts to capture both effectiveness and efficiency. We implemented this framework to evaluate 14 state-of-the-art LLMs on exploiting 36 real CVEs using 2 exploit tools (Metasploit and GitHub PoC) and a correction loop that enables LLMs to correct their previous exploitation attempts. Experimental results reveal variation in the behavior of different LLMs, using both exploitation tools, with closed-source models generally outperforming open-source ones.

## CCS Concepts

* Security and privacy → Penetration testing; * Information systems → Language models.

## Keywords

Large Language Models, Offensive Cybersecurity, CVE Exploitation

## ACM Reference Format:

Mohamed Amine El yagouby, Abdelkader Lahmadi, Mehdi Zakroum, Olivier Festor, and Mounir Ghogho. 2025. LLM-CVX: A Benchmarking Framework for Assessing the Offensive Potential of LLMs in Exploiting CVEs. In *Proceedings of the 2025 Workshop on Artificial Intelligence and Security (AISec '25), October 13-17, 2025, Taipei, Taiwan*. ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/3733799.3762978

## 1 Introduction

Advances in generative AI, particularly Large Language Models (LLMs), have been rapidly adopted by cyber-attackers to automate attacks. Since the launch of ChatGPT at the end of 2022, the volumes of phishing attacks have increased by 4151% [3], and bot traffic constitutes nearly one third of all web traffic [20]. Meanwhile, malicious LLMs such as WormGPT [1] and FraudGPT [12] are being developed and sold on the dark web for subscription fees between $200 per month and $1700 per year, with more than 3000 confirmed sales of FraudGPT alone [12]. The sellers claim that these tools can draft phishing emails and web pages, scan for website vulnerabilities, and automate attack scripts, which could enable even less technically skilled cyber-attackers to launch effective attacks.

Evaluating the offensive potential of LLMs is therefore a critical step toward anticipating these emerging threats. As these models gain proficiency in code generation, tool usage, and reasoning about vulnerabilities, their potential role in exploit development becomes increasingly concerning.

Current LLMs' evaluation methodologies in offensive security are limited in both scope and design. Most of them rely on synthetic or simplified tasks, such as Capture The Flag (CTF) challenges [26], which fail to capture the relevance to real vulnerable systems, and none of these methods evaluate LLMs independently on specific prompting strategies and exploitation tools used. Instead, they typically depend on fixed prompts and tools, making it difficult to understand whether the exploitation success is due to the LLM's capabilities itself or the prompting strategy and tools employed. In addition, they neglect the number of attempts required to achieve

<figure><img src="image_4.png" /></figure>

This work is licensed under a Creative Commons Attribution 4.0 International License. AISec '25, Taipei, Taiwan

© 2025 Copyright held by the owner/author(s).
ACM ISBN 979-8-4007-1895-3/25/10
https://doi.org/10.1145/3733799.3762978
successful exploitation, which is an important factor that directly influences the attack efficiency, as real-world attackers prioritize stealth and speed. In our scenario, an LLM that succeeds in fewer attempts can be considered to pose a greater threat.

To address these limitations, we introduce **LLM-CVX** (LLMs for CVE Exploitation benchmarking framework), a novel framework that provides controlled evaluation across various prompting strategies and exploitation tools. In this framework vulnerable environments are built automatically and then destroyed after all LLMs have finished attempting to exploit these systems, with the possibility of making multiple attempts up to a fixed maximum.

Two outcomes are then provided by our framework. First, a benchmark ranking LLMs with three existing metrics that assess both effectiveness and efficiency [41], Success Rate (SR), Average Task Completion Attempts (AvgTCA), and Success Efficiency (SE). The second outcome is an analysis of the performance of these LLMs across MITRE ATT&CK tactics and techniques, which are mapped to corresponding CVEs. We implemented LLM-CVX to evaluate 14 state-of-the-art LLMs in exploiting 36 CVEs using two exploit tools, Metasploit and GitHub PoCs.

Specifically, the key contributions of this work are as follows:

* We introduce LLM-CVX, a novel, extensible, and fully automated framework to evaluate the potential of LLMs in exploiting known CVEs within controlled, isolated vulnerable environments.

* We designed this framework to be extensible towards exploitation tools and prompting strategies, thereby enabling fair and systematic comparisons across configurations that influence LLM performance.

* This framework enables LLMs to perform multiple exploitation attempts to assess both their effectiveness and efficiency.

* We implement this framework to evaluate 14 LLMs on 36 real-world CVEs, and analyze their performance across corresponding MITRE ATT&CK tactics and techniques.

While our framework provides a controlled and reproducible setting that ensures fair benchmarking across LLMs, along with different prompting strategies and exploit tools, it is limited to one CVE at a time exploitation on an isolated target machine. Multi-stage attack chains and interconnected environments remain out of scope and are left for future work.

The rest of this paper is structured as follows. Section 2 provides the necessary background and motivation for our work. Section 3 reviews related works. Section 4 presents our proposed benchmarking framework, followed by experimental setup and results in Sections 5 and 6. Finally, we discuss key findings, future work, and ethical considerations in Sections 7 and 8, and conclude in Section 9.

## 2 Background and Motivation

This section presents the foundational concepts used in this work, including LLMs, offensive cybersecurity, CVEs with exploit tools and MITRE ATT&CK framework, and further explains the motivation for this research, particularly the use of LLMs in offensive cybersecurity and the rise of malicious LLMs.

### 2.1 Large Language Models (LLMs)

Large Language Models have demonstrated advanced capabilities in natural language understanding [8], code generation [10], reasoning [24], and tool usage [31], establishing their utility across related computational tasks [7], but also raise concerns about their misuse [37]. These models vary from close to open-source, and the latter are particularly concerning in this context due to their ability to be further fine-tuned on malicious contexts [22].

**Emergence of Malicious LLMs:** Malicious LLMs such as **WormGPT** [1] and **FraudGPT** [12] are being actively developed and sold on the dark web. Table 1 compares the advertised characteristics and technical functionalities. Their developers explicitly claim that both of these LLMs have no ethical guardrails, can assist in malware creation, and facilitate different threat vectors: WormGPT is often associated with business email compromise (BEC) and malware generation, while FraudGPT assists in creating tools for online fraud and phishing. Their subscription fees range from $100/month and private builds up to $5,000 for WormGPT [14], and prices between $300–$1,700 for FraudGPT, which recorded over 3,000 confirmed sales [11], demonstrating significant adoption among cybercriminals of these tools. The ease of use and low barrier to entry of such tools make them accessible even to less technical threat actors, which risks raising the scale and reach of cyberattacks and enabling further automation of these attacks.

<table><caption>Table 1: Comparison of WormGPT and FraudGPT</caption><thead><tr><th></th><th>WormGPT</th><th>FraudGPT</th></tr></thead><tbody><tr><th>Base Model</th><td>GPT-J (2021) [14]</td><td>Fine-tuned GPT [14]</td></tr><tr><th>Pricing</th><td>$100/mo or $5K private build [14]</td><td>$300/mo or $1.7K/yr [14]</td></tr><tr><th>Confirmed Sells</th><td>3000 [11]</td><td>—</td></tr><tr><th>Main Threats</th><td>BEC, malware, phishing [11, 14]</td><td>fraud site generation, malware [11, 14]</td></tr></tbody></table>

### 2.2 Offensive Cybersecurity

The offensive part of cybersecurity also known as red teaming [9] plays a critical role in enhancing its defensive counterpart by simulating attacks under ethical and controlled conditions [4, 13], it helps identify system weaknesses before real attackers can exploit them [5] or understand their exploitation in the wild [21]. In our case, evaluating the offensive potential of LLMs allows us to anticipate how they could be misused and to understand some existing cases of misuse as described previously in Section 2.1, thereby informing with these new threats to help design more effective detection and mitigation strategies.

### 2.3 Common Vulnerabilities and Exposures

The Common Vulnerabilities and Exposures (CVE) system<fnref n="1" />, launched in 1999 by the Mitre corporation<fnref n="2" />, is an international standard for cataloging known cybersecurity vulnerabilities, serving as a foundation for their disclosure and patching [25]. Each CVE entry contains

<footnote n="1">¹https://www.cve.org/</footnote>

<footnote n="2">²https://www.mitre.org/</footnote>
a unique identifier along with metadata such as a description of the vulnerability, affected systems, and severity scores like the *Common Vulnerability Scoring System (CVSS)* [27], which ranges from 0 to 10, corresponding to four severity levels (*LOW* (0.1-3.9), *MEDUIM* (4-6.9), *HIGH* (7-8.9) and *CRITICAL* (9-10)) to reflect the complexity and impact of the vulnerability.

Some CVEs have publicly available exploits, typically published only after the vulnerability is patched and the original developers release a secure version of the affected system to prevent potential harm, with only 2 days as the median time from patch release to first exploit publication [19]. These public exploits can be hosted on platforms like *GitHub as proof-of-concepts (PoCs)* [35, 40], containing exploit codes to demonstrate vulnerabilities, and often developed and published by independent researchers, or integrated as modules in penetration testing frameworks such as *Metasploit*<fnref n="3" />, which is an open source modular framework to test and execute exploits against remote target systems. Alternatively, new exploits can be created by developing custom code, which is used for CVEs that do not yet have public exploits, typically more recent ones, also called *Zero-Day vulnerabilities* [23]. In this work, these are referred to as *exploit tools*.

## 2.4 MITRE ATT&CK Framework

The MITRE ATT&CK framework<fnref n="4" />, launched in 2015, is a structured knowledge base that categorizes tactics and techniques based on behaviors observed in real-world cyber intrusions. Where each tactic and technique is represented by a unique identifier (ID), a name, a description, mitigation strategies and other details. Tactics define the high-level goals or stages of an attack, such as initial access, execution, and privilege escalation, and techniques specify the concrete methods adversaries employ to achieve these goals, such as process injection or network denial of service [33]. In this work, we mapped the CVEs to their corresponding tactics and techniques, to enable an analysis of which aspects different LLMs are most capable of covering, as detailed in Section 5.1.

## 3 Related Works

Several recent studies have proposed methodologies to evaluate the capabilities of LLMs to exploit vulnerabilities. Table 2 compares and summarizes the key characteristics of these studies.

**Real-World vs. Synthetic Vulnerabilities** A major axis of differentiation in these works is the vulnerabilities on which LLMs are evaluated to exploit. As discussed previously in Section 2.3, real-world vulnerabilities corresponding to CVEs, represent security flaws discovered in live, deployed software. In contrast, synthetic vulnerabilities, often taken from CTF challenges [26] are artificially designed for training purposes. For instance, CYBENCH [43], CTFLLM [32] and HackSynth [28] evaluate LLMs on these CTF challenges, involving areas like binary exploitation, cryptography, and reverse engineering. While these challenges are useful for structured evaluation, they lack the contextual relevance to real-world discovered vulnerabilities such as CVEs. To the best of our knowledge, only three of the nine studies, CVE-Bench [44], LLMExp1Day [15] and AUTOPENBENCH [17], explicitly use CVEs.

**LLMs Evaluated and Performance Gaps** These studies test both closed and open-source LLMs, revealing a large performance gap, with proprietary closed models often outperforming open-source ones across nearly all works. Recent GPT models [2], such as GPT-4 and GPT-4o, were the most effective, with success rates reaching up to 87%, unlike open-source LLaMA models [34], which were less effective, with 0% in success rate.

**Exploit Tools and Prompting Strategies Dependency** Different studies rely on specific exploit tools tied to their evaluation setup. For example, AutoAttacker [38] and AUTOPENBENCH [17] leverage Metasploit, while AutoAttacker employs bash and PowerShell scripts. Additionally LLMExp1Day [15] uses Python and shell scripts, and CYBENCH [43] relies on bash tools within a Kali Linux environment.

Prompting strategies also vary, ReAct [42] prompting, used in LLMHackWeb [16], LLMExp1Day [15] and CYBENCH [43], enables the model to alternate between reasoning and tool usage. Others like AutoAttacker [38] and HackSynth [28] rely on agent-based promptings, where specific roles are assigned to these agents, like planning, summarizing previous steps, and acting. Additionally, CTFLLM [32] tested both fully automated exploitation and Human-in-the-Loop (HITL) methods. These variations significantly influence the results. Additionally, CVE-Bench [44] evaluates LLM agents rather than raw LLMs, where each agent couples a fixed prompting strategy with specific exploit tools such as sqlmap or bash, and all agents rely on a single backbone LLM for generation.

However, all these methodologies are overly reliant on specific exploit tools and prompting strategies, which limits generalization, and makes it unclear whether the effectiveness is due to LLM's reasoning capabilities or the tools and promptings in use.

**Lack of Explicit Efficiency Metrics** Another critical limitation in existing works is the lack of explicit efficiency measurement in terms of the number of attempts required until the first successful exploitation attempt. Instead, benchmarks such as CYBENCH [43] and CVE-Bench [44] rely on implicit metrics, namely the average number of tokens and the average inference time consumed by an LLM during exploitation attempts across all tasks. While reporting these raw signals is informative, is not tied to outcomes, as a model can spend many tokens and minutes and still fail, with logs unable to distinguish wasted effort from efficient success. Also they are not standardized across LLMs, since tokenization schemes differ for each LLM, and raw runtime is affected by hardware or API latency. Additionally, CVE-Bench [44] reports monetary cost in dollars, but this is vendor-dependent, volatile to pricing changes, and inapplicable to open-source LLMs that can be executed locally.

Other studies report overall average success rates over multiple retries but also do not count how quickly different LLMs achieve the first success, which is an important factor for practical relevance, given that, in real-world scenarios, stealth and speed are vital, as multiple failed attempts increase the chance of detection. Therefore, an LLM that requires fewer attempts to exploit vulnerabilities can be considered more efficient and dangerous.

**LLMs' autonomy in exploitation has been criticized.** Fang et al. propose that LLMs can autonomously exploit websites in LLMHackWeb [16] and One-Day vulnerabilities in LLMExp1Day [15]. In these setups, the LLM is prompted (using React [42]) to plan its next move, retrieves documentation (in LLMExp1Day) or browses

<footnote n="3">³https://www.metasploit.com/</footnote>

<footnote n="4">⁴https://attack.mitre.org/</footnote>
<table><caption>Table 2: Summary and comparison of existing works on evaluating LLMs in exploiting vulnerabilities focusing on: prompting and exploit tool independence, CVE usage, and efficiency</caption><thead><tr><th>Study</th><th>#LLMs</th><th>#Vuls</th><th>Prompting</th><th>Exploit Tools</th><th>P. Ind.[5]</th><th>ET. Ind.[6]</th><th>Efficiency</th><th>CVE based</th></tr></thead><tbody><tr><td>LLMHackWeb*[16]</td><td>10</td><td>15</td><td>ReAct</td><td>Shell, Python</td><td>No</td><td>No</td><td>No</td><td>No</td></tr><tr><td>LLMExp1Day*[15]</td><td>10</td><td>15</td><td>ReAct</td><td>Shell, Python</td><td>No</td><td>No</td><td>No</td><td>Yes</td></tr><tr><td>AUTOATTACKER[38]</td><td>4</td><td>14</td><td>Agent planning loop</td><td>Metasploit, Bash, PS</td><td>No</td><td>No</td><td>No</td><td>No</td></tr><tr><td>CTFLLM[32]</td><td>6</td><td>26</td><td>HITL + Retry loop</td><td>CLI, RE tools</td><td>No</td><td>No</td><td>No</td><td>No</td></tr><tr><td>AUTOPENBENCH[17]</td><td>6</td><td>33</td><td>CoALA</td><td>Metasploit, Hydra, Nmap</td><td>No</td><td>No</td><td>No</td><td>Yes</td></tr><tr><td>HackSynth[28]</td><td>8</td><td>200</td><td>Planner, Summarizer agents</td><td>Linux shell tools</td><td>No</td><td>No</td><td>No</td><td>No</td></tr><tr><td>CYBERSECEVAL 2[6]</td><td>5</td><td>15</td><td>Direct prompt to exploit</td><td>Python, SQL, JS</td><td>No</td><td>No</td><td>No</td><td>No</td></tr><tr><td>CYBENCH [43]</td><td>8</td><td>40</td><td>ReAct/Reflexion style</td><td>Bash in Kali</td><td>No</td><td>No</td><td>Tokens, time</td><td>No</td></tr><tr><td>CVE-Bench [44]</td><td>3 agents</td><td>40</td><td>Fixed per agent</td><td>Fixed per agent</td><td>No</td><td>No</td><td>Tokens, time</td><td>Yes</td></tr><tr><td>LLM-CVX (Ours)</td><td>14</td><td>36</td><td>Correction loop</td><td>Metasploit, Github PoC</td><td>Yes</td><td>Yes</td><td>AvgTCA, SE</td><td>Yes</td></tr></tbody></table>

the web (in LLMHackWeb), and then executes exploitation actions like crafting payloads or running shell commands. Additionally, in LLMHackWeb, LLMs have access to websites' backend source code, while LLMExp1Day used CVEs released after all tested LLMs' knowledge cut-offs to propose autonomy in LLMs' behaviors. These two works report high success rates of certain LLMs (73% and 87%, respectively). However, they were critically reviewed by Chris Rohlf [29, 30], who challenged the claims of true autonomy in LLMs' exploitation. For LLMHackWeb, he argued that access to source code reduces the task to static analysis rather than active exploitation. For LLMExp1Day, he showed that for 11 out of 15 CVEs, public exploits could be found via "the first Google search result", questioning whether LLMs truly discovered or just retrieved existing exploits via the browsing tools that they had access to in this setup. These critiques highlight the need for more rigorous evaluations of LLMs' true capabilities in this context.

# 4 Methodology

The design of a new benchmarking framework that addresses the gaps in existing works described above in Section 3, requires addressing the following problem:

**Problem Statement.** Given a dataset of CVEs, where each CVE entry includes its identifier and description, and a list of candidate LLMs, the objective is to automatically evaluate each LLM on exploiting the corresponding vulnerable systems of each CVE. The evaluation must consider three key constraints:

(1) **Automatic Vulnerable Environments:** The framework should automatically build the vulnerable environments corresponding to each CVE in the dataset to provide a consistent testbed for LLM exploitation attempts.

(2) **Independence from Prompting and Exploit Tool:** It should also be extensible to use multiple prompting strategies and exploitation tools, which avoid being biased toward any specific ones.

(3) **Efficiency Capturing:** The evaluation should consider not only whether the LLM succeeds, but also how efficiently it does so, measured by the number of attempts taken to

succeed within a predefined limit, to reflect practical performance assessment.

## 4.1 LLM-CVX Benchmarking Framework

To meet these requirements, we propose a fully automated benchmarking framework for evaluating LLMs on CVE exploitation tasks. A key feature of the framework is its extensible design, allowing evaluations to be conducted independently of the prompting strategy or exploitation tool used.

As illustrated in Figure 1, the benchmark operates with two machines: (1) a target machine where vulnerable systems corresponding to specific CVEs are deployed, and (2) an AI-assisted attack machine responsible for interacting with the LLM and executing the exploitation process. These two machines are interconnected on an isolated network, eliminating the risk of affecting external systems or networks.

For each CVE in the dataset, the framework automatically executes the following sequence of steps:

(1) **Environment Setup:** A vulnerable environment is automatically built on the target machine using a pre-configured setup (such as Docker builds<fnref n="5" />), to ensure reproducible and standardized vulnerable environments across different CVEs.

(2) **LLM Initialization:** The LLM is initialized along with a prompting strategy and exploit tools.

(3) **Exploitation Loop:** The LLM can attempt to exploit the vulnerable system up to a predefined maximum number of attempts, following an abstract workflow that must be implemented according to the exploit tool and prompting strategy in use, where each attempt consists of:

(a) **Generation** of an exploit adapted to the specified tool (such as Metasploit commands, Commands to execute a GitHub PoC, or custom code).

(b) **Execution** of the generated exploit from the AI-assisted attacking machine against the target system.

<footnote>[5] Prompting Independence in evaluating LLMs.</footnote>

<footnote>[6] Exploit Tool Independence in evaluating LLMs.</footnote>

<footnote>* Acronyms were assigned to these two studies for clarity and</footnote>

<footnote>readability.</footnote>

<footnote n="5">⁵Docker is a platform that automate the deployment of applications by packaging them into isolated containers, which include all necessary dependencies and run consistently across environments.</footnote>
<figure><img src="image_5.png" /><figcaption>Figure 1: Illustration of LLM-CVX Benchmarking Framework</figcaption></figure>

(c) **Success Check** to automatically determine whether the exploit was successful or if the maximum number of attempts has been reached. If neither condition is met, the process returns to the (a) **generation** phase, and this time, additional input data, such as the LLM's previous response and the corresponding execution output, can be used to help the LLM refine its next attempt.

(4) **Result Recording and Cleanup**: Upon successful exploitation or exhaustion of attempts, results, and artifacts containing all LLM responses, and the execution logs are all recorded, and the vulnerable environment is destroyed to prepare for the next CVE evaluation.

Two outcomes are derived from the aggregated results of each LLM tested across the complete CVE dataset. First, a benchmark table ranking LLMs along with prompting strategy and exploit tools used, with three metrics: Success Rate (SR), Average Task Completion Attempts (AvgTCA), and Success Efficiency (SE) to provide quantitative assessment of both the effectiveness and efficiency of LLMs in performing exploitation tasks, as detailed in Section 4.2. The second outcome is a performance analysis across MITRE ATT&CK tactics and techniques corresponding to the tested CVEs. This correspondence is established by mapping each CVE in the dataset to its associated tactics and techniques, which provides insights into the performance of different LLMs and highlights areas where automated exploitation using LLMs may be more effective.

Finally, the framework's extensible design allows easy integration of new LLMs, CVEs, prompting strategies, and exploitation tools, facilitating comprehensive performance comparisons across different configurations of LLMs' offensive potential in exploiting CVEs.

## 4.2 Evaluation Metrics

We used three existing evaluation metrics, introduced in previous work [41], to evaluate LLMs on binary-outcome tasks across multiple attempts. These metrics are particularly well-suited in our setting, where each exploitation attempt either succeeds or fails, and LLMs are given multiple opportunities to complete the task. These metrics are used in our context as follows:

Let $MaxA \ge 1$ denote the maximum number of allowed attempts per CVE exploitation.

(1) **Success Rate (SR)**

$$SR = \frac{\text{Exploited CVEs}}{\text{Total CVEs}}$$

SR quantifies the proportion of CVEs that the LLM successfully exploited within the allowed number of attempts. A higher SR indicates greater overall effectiveness in achieving the exploitation objective.

(2) **Average Task Completion Attempts (AvgTCA)**

$$\text{AvgTCA} = \frac{1}{\text{Exploited CVEs}} \sum TCA$$

TCA (Task Completion Attempts) counts the number of attempts taken to successfully exploit a CVE. AvgTCA measures the average number of attempts required for successful exploitation. Lower values reflect more efficient exploitation behavior.

(3) **Success Efficiency (SE)**

$$SE = \frac{SR}{\text{AvgTCA} \left( \frac{\text{AvgTCA}-1}{\text{MaxA}-1} \right)}$$

SE is a composite metric that balances effectiveness and efficiency. It is bounded in the range [0, SR], reaching its maximum when all exploited CVEs are completed on the first attempt (AvgTCA = 1), and decreasing as the average
number of attempts increases. The exponent acts as a normalization factor, measuring how far the LLM's AvgTCA deviates from the ideal value of 1 (perfect efficiency) normalized by the maximum possible deviation (MaxA - 1) which makes SE scales with respect to MaxA and penalizes inefficient success.

# 5 Experimental Setup

This section describes the set of CVEs and LLMs evaluated in our study, as well as the exploit tools and prompting strategies employed. All code and data employed in the experiments are publicly available<fnref n="6" />. The repository provides (1) scripts for configuring a target Ubuntu virtual machine with the vulnerable systems corresponding to the evaluated CVEs, together with the AI-assisted attacker Kali Linux virtual machine pre-configured with the required dependencies and exploit tools; and (2) code for orchestrating the benchmarking framework from the host machine. The latter includes, for each CVE, running the vulnerable environment on the target VM, and for each LLM, generating and executing exploits from the attacker VM towards the target VM, then saving all results and artifacts (namely, the LLMs' responses in each attempt containing exploitation commands along with their corresponding execution outputs); and (3) code for producing a benchmark ranking table of LLMs, based on the evaluation metrics described in Section 4.2, together with their performance analysis across MITRE ATT&CK tactics and techniques, as specified by our benchmarking framework in Section 4.1.

## 5.1 CVEs Dataset

We collected a dataset of 36 CVEs, each identified by its CVE-ID and accompanied by a description. For each CVE, we used its corresponding vulnerable systems provided by the Vulhub project<fnref n="7" />, which offers Docker images that can be built using standardized Dockerfiles and later destroyed, as specified earlier by the framework design in Section 4.1. Figure 2 shows the CVSS v3.1 severity distribution of the selected CVEs, where most CVEs fall into the Critical (65.7%) and High (28.6%) severity categories, which reflects the impact of these vulnerabilities. All selected CVEs have publicly available exploits, which makes the goal of this experiment to evaluate the LLMs' ability to automatically exploit these CVEs using existing tools (detailed in Section 5.3), rather than assessing their capability to generate new exploits for undisclosed vulnerabilities (CVEs with no public exploits).

Each CVE in the dataset was manually mapped to relevant MITRE ATT&CK tactics and techniques by analyzing its description, exploitation method, and behavior. This included reviewing public exploits and vendor advisories to identify the corresponding adversary actions. As shown in Figures 3 and 4, the most frequent tactics are Initial Access (TA0001), Execution (TA0002), and Privilege Escalation (TA0004), often involving techniques such as Command and Scripting Interpreter (T1059), Process Injection (T1055), and Exploitation of Public-Facing Application (T1190). These techniques enable attackers to execute arbitrary code, bypass security controls, and escalate privileges, which are

<figure><img src="image_6.png" /><figcaption>Figure 2: Distribution of CVSS v3.1 severity scores among the selected CVEs. Most are Critical (65.7%) or High (28.6%), categories that denote severe security impact.</figcaption></figure>

key steps in achieving full system compromise. Together, they illustrate how the selected CVEs are relevant to practical scenarios.

<figure><img src="image_7.png" /><figcaption>Figure 3: Distribution of CVEs by associated MITRE ATT&CK tactics. The most common tactics include Privilege Escalation, Execution, and Initial Access.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Figure 4: Distribution of CVEs by MITRE ATT&CK techniques. Process Injection (T1055) and Command and Scripting Interpreter (T1059) are among the most prevalent techniques.</figcaption></figure>

<footnote n="6">⁶https://gitlab.inria.fr/resist/llm-cvx</footnote>

<footnote n="7">⁷https://github.com/vulhub/vulhub</footnote>
The CVEs used and their mappings are provided in the Appendix in Tables 9, 7 and 8.

## 5.2 LLMs Evaluated

We selected 14 state-of-the-art LLMs for evaluation on this CVE dataset, including 8 proprietary (closed-source) and 6 open-source models. As detailed in Table 3, these models come from various providers and may differ in architecture and training data, which can potentially result in varied behavior across exploitation tasks.

All LLMs were accessed via the OpenRouter API<fnref n="8" /> using their default generation parameters, such as *temperature* and *top-p*, to ensure consistency and avoid bias from manual tuning.

<table><caption>Table 3: Details about the LLMs tested with Source, Name, Release Date, and Developer.</caption><thead><tr><th>Source</th><th>LLM</th><th>R. Date</th><th>Developer</th></tr></thead><tbody><tr><td rowspan="8">Closed</td><td>claude-3.7-sonnet</td><td>2025-06</td><td>Anthropic</td></tr><tr><td>gemini-2.0-flash-001</td><td>2024-12</td><td>Google</td></tr><tr><td>gemini-2.5-pro-preview</td><td>2025-06</td><td>Google</td></tr><tr><td>gpt-4.1</td><td>2024-11</td><td>OpenAI</td></tr><tr><td>gpt-4o</td><td>2024-05</td><td>OpenAI</td></tr><tr><td>gpt-4o-mini</td><td>2025-06</td><td>OpenAI</td></tr><tr><td>grok-3-beta</td><td>2024-12</td><td>xAI</td></tr><tr><td>grok-3-mini-beta</td><td>2025-05</td><td>xAI</td></tr><tr><td rowspan="6">Open</td><td>deepseek-r1</td><td>2024-12</td><td>DeepSeek</td></tr><tr><td>deepseek-r1-distill-llama-70b</td><td>2025-04</td><td>DeepSeek</td></tr><tr><td>llama-3.3-70b-instruct</td><td>2025-06</td><td>Meta</td></tr><tr><td>mistral-large-2407</td><td>2024-07</td><td>Mistral</td></tr><tr><td>codestral-2501</td><td>2024-05</td><td>Mistral</td></tr><tr><td>qwen-2.5-coder-32b-instruct</td><td>2025-05</td><td>Alibaba</td></tr></tbody></table>

## 5.3 Exploit Tools, Prompting and Implementation

Two exploit tools were used with the LLMs: *Metasploit* and *GitHub PoCs*. As all selected CVEs have publicly available exploits, implemented as Metasploit modules executable via *Msfconsole*, and also provided as PoCs on GitHub containing exploitation code. We selected only those PoCs implemented in Python to unify the execution environment. These PoCs were pulled and stored locally, and some were slightly modified, primarily by editing their *README* files to include clear execution instructions and by adjusting the main code to print standardized success or failure flags upon successful exploitation. These modifications did not alter the core logic or functionality of the exploits.

Implementing the abstract exploitation workflow for each of these tools to integrate them into our benchmarking framework, involves implementing the three core functionalities of the abstract exploitation workflow of the benchmarking framework, described earlier in Section 4.1. As illustrated in Figure 5, each of these functionalities is implemented as follows:

(1) **Exploit Generation:** With a prompting strategy that enables LLMs to correct their previous attempts, which we

<figure><img src="image_9.png" /><figcaption>Figure 5: Exploitation workflow implementation for Metasploit and GitHub PoC with the correction loop prompting strategy</figcaption></figure>

refer to in this paper as the *correction loop*. This loop starts with an initial attempt, in which the LLM receives a prompt containing the CVE details, including the CVE ID, description, and environment information (such as target and local IPs, ports, etc.). For GitHub PoCs, additional information is included: the local path to the PoC and its README file. Based on this input, the LLM generates the initial exploit commands. If the first attempt fails, the next prompt includes both the *Previous LLM Response* (PLR) and the corresponding *Execution Output* (PEO) to help the LLM correct its earlier response.

(2) **Exploit Execution:** After each generation, exploit commands are extracted from the responses and executed as follows:

<footnote n="8">⁸https://openrouter.ai</footnote>
* **For Metasploit:** Commands are sent over a TCP socket to a server, which maintains a persistent msfconsole instance to avoid the high startup cost of launching a new instance on every attempt, which takes in our setup more than 10 seconds.

* **For GitHub PoC:** Commands are executed directly in the local terminal environment.

(3) **Check Exploit Success:** This is implemented differently for each exploit tool across all CVEs, enabling a unified and automatic check of whether an exploit attempt was successful:

* **For Metasploit:** In cases of CVEs that open sessions to the target machine after a successful exploitation, two additional commands are executed after the exploit commands: *sessions* (to list active sessions) and *sessions -K* (to terminate any existing sessions), to allow the session information to be printed in the execution output. For other CVEs, where exploitation does not result in an open session, we collected success patterns through manual testing. These patterns consist of specific strings that appear in the output upon successful exploitation, or example, "Found password: admin:supersecret123" for an information disclosure CVE, or "File written: /var/www/html/shell.jsp" for a CVE allowing arbitrary file writing. Finally, all these patterns are checked in the execution output to determine whether the exploit succeeded or failed.

* **For GitHub PoC:** Simply checks for a success flag in the execution output, which was standardized in pre-modified exploit codes.

# 6 Experimental Results

This section presents the benchmark results and LLMs' performance analysis by MITRE ATT&CK tactics and techniques, obtained by implementing our LLM-CVX benchmarking framework on the CVE dataset and LLM setup described in Section 5.

## 6.1 Benchmark

Across both exploit tools, GPT-4.1 consistently outperformed all other LLMs. As shown in Table 4, it achieved the highest SR and SE scores with both GitHub PoCs (SR: 0.81, SE: 0.56) and Metasploit (SR: 0.86, SE: 0.53). In contrast, Deepseek-r1 has the lowest performance across both tools, with GitHub PoCs (SR: 0.31, SR: 0.21) and Metasploit (SR: 31 A, SR: 0.15).

These results also show that closed-source LLMs dominated the ranking, with seven models among the top ten based on the SE ranking. Nonetheless, certain open-source models, such as *Codestral-2501* and *Qwen-2.5-Coder-32B-Instruct*, also performed well. For example, *Codestral-2501* ranked 4th with GitHub PoCs and 6th with Metasploit. On average, these closed-source LLMs slightly outperformed their open-source counterparts, both in SE (0.38 vs. 0.37) and Success Rate (SR) (0.60 vs. 0.57), with a similar standard deviation in the average values of the three metrics across LLMs, as presented in Table 5.

**LLM performance by exploit tool.** As shown in Table 6, LLMs achieved the same average SR of 0.59 using either GitHub PoC or Metasploit. However, they were more efficient with GitHub PoC,

<table><caption>Table 4: Benchmarking results</caption><thead><tr><th>Rank</th><th>LLM</th><th>E. Tool</th><th>SR</th><th>AvgTCA</th><th>SE</th></tr></thead><tbody><tr><td>1</td><td>gpt-4.1</td><td>Github PoC</td><td>0.81</td><td>2.55</td><td><strong>0.56</strong></td></tr><tr><td>2</td><td>gpt-4.1</td><td>Metasploit</td><td><strong>0.86</strong></td><td>2.87</td><td>0.53</td></tr><tr><td>3</td><td>claude-3.7-sonnet</td><td>Github PoC</td><td>0.78</td><td>2.68</td><td>0.51</td></tr><tr><td>4</td><td>codestral-2501</td><td>Github PoC</td><td>0.67</td><td>2.29</td><td>0.51</td></tr><tr><td>5</td><td>gpt-40</td><td>Github PoC</td><td>0.72</td><td>2.58</td><td>0.50</td></tr><tr><td>6</td><td>codestral-2501</td><td>Metasploit</td><td>0.78</td><td>2.79</td><td>0.49</td></tr><tr><td>7</td><td>claude-3.7-sonnet</td><td>Metasploit</td><td>0.72</td><td>2.69</td><td>0.48</td></tr><tr><td>8</td><td>gemini-2.0-flash-001</td><td>Github PoC</td><td>0.64</td><td>2.52</td><td>0.45</td></tr><tr><td>9</td><td>qwen-2.5-coder-32b-instruct</td><td>Metasploit</td><td>0.67</td><td>2.67</td><td>0.44</td></tr><tr><td>10</td><td>gemini-2.0-flash-001</td><td>Metasploit</td><td>0.67</td><td>2.67</td><td>0.44</td></tr><tr><td>11</td><td>grok-3-beta</td><td>Github PoC</td><td>0.67</td><td>2.67</td><td>0.44</td></tr><tr><td>12</td><td>llama-3.3-70b-instruct</td><td>Metasploit</td><td>0.67</td><td>2.71</td><td>0.44</td></tr><tr><td>13</td><td>llama-3.3-70b-instruct</td><td>Github PoC</td><td>0.61</td><td>2.50</td><td>0.43</td></tr><tr><td>14</td><td>gpt-40</td><td>Metasploit</td><td>0.69</td><td>2.88</td><td>0.42</td></tr><tr><td>15</td><td>mistral-large-2407</td><td>Github PoC</td><td>0.64</td><td>2.70</td><td>0.42</td></tr><tr><td>16</td><td>grok-3-beta</td><td>Metasploit</td><td>0.64</td><td>2.78</td><td>0.40</td></tr><tr><td>17</td><td>qwen-2.5-coder-32b-instruct</td><td>Github PoC</td><td>0.56</td><td>2.55</td><td>0.39</td></tr><tr><td>18</td><td>mistral-large-2407</td><td>Metasploit</td><td>0.61</td><td>2.91</td><td>0.37</td></tr><tr><td>19</td><td>deepseek-r1-distill-llama-70b</td><td>Github PoC</td><td>0.50</td><td>2.67</td><td>0.33</td></tr><tr><td>20</td><td>gemini-2.5-pro-review</td><td>Github PoC</td><td>0.47</td><td>2.88</td><td>0.29</td></tr><tr><td>21</td><td>gpt-40-mini</td><td>Github PoC</td><td>0.44</td><td>2.75</td><td>0.29</td></tr><tr><td>22</td><td>deepseek-r1-distill-llama-70b</td><td>Metasploit</td><td>0.53</td><td>3.26</td><td>0.27</td></tr><tr><td>23</td><td>grok-3-mini-beta</td><td>Github PoC</td><td>0.39</td><td>2.71</td><td>0.25</td></tr><tr><td>24</td><td>gemini-2.5-pro-review</td><td>Metasploit</td><td>0.50</td><td>3.56</td><td>0.22</td></tr><tr><td>25</td><td>deepseek-r1</td><td>Github PoC</td><td><strong>0.31</strong></td><td>2.64</td><td>0.21</td></tr><tr><td>26</td><td>gpt-40-mini</td><td>Metasploit</td><td>0.33</td><td>3.33</td><td>0.17</td></tr><tr><td>27</td><td>grok-3-mini-beta</td><td>Metasploit</td><td>0.33</td><td>3.42</td><td>0.16</td></tr><tr><td>28</td><td>deepseek-r1</td><td>Metasploit</td><td><strong>0.31</strong></td><td>3.36</td><td><strong>0.15</strong></td></tr></tbody></table>

<table><caption>Table 5: Closed vs Open Source LLMs Performance</caption><thead><tr><th>Source</th><th>SR</th><th>AvgTCA</th><th>SE</th></tr></thead><tbody><tr><td>Closed-Source</td><td><strong>0.60</strong> ± 0.17</td><td>2.85 ± 0.31</td><td><strong>0.38</strong> ± 0.13</td></tr><tr><td>Open-Source</td><td>0.57 ± 0.14</td><td>2.75 ± 0.30</td><td>0.37 ± 0.11</td></tr></tbody></table>

<table><caption>Table 6: Exploit Tools Average Performance Comparison</caption><thead><tr><th>Exploit Tool</th><th>SR</th><th>AvgTCA</th><th>SE</th></tr></thead><tbody><tr><td>GitHub PoC</td><td><strong>0.59</strong> ± 0.15</td><td>2.62 ± 0.14</td><td><strong>0.40</strong> ± 0.11</td></tr><tr><td>Metasploit</td><td><strong>0.59</strong> ± 0.17</td><td>2.99 ± 0.32</td><td>0.36 ± 0.13</td></tr></tbody></table>

with a higher SE score (0.40 vs. 0.36), due to requiring fewer attempts across successful exploitations (2.62 vs. 2.99). GitHub PoC also showed lower variation in SE across LLMs (standard deviation of 0.11 vs. 0.13 for Metasploit). Additionally, among the LLMs tested with both tools, 12 performed better with GitHub PoC, while only LLaMA-3.3-70B-Instruct and Qwen-2.5-Coder-32B-Instruct performed slightly better with Metasploit, with scores, though they still required more attempts than with GitHub PoC.

## 6.2 LLMs Efficiency by MITRE ATT&CK

Figure 6 presents the SE scores of the top three (LLM, exploit tool) pairs in the benchmark ranking table (Table 4), distributed across
<figure><img src="image_10.png" /><figcaption>Figure 6: Top 3 LLM SE scores distribution across MITRE ATT&CK Tactics/Techniques</figcaption></figure>

MITRE ATT&CK tactics and techniques, which shows that the most frequent tactics and techniques among the CVEs tested (as detailed in Section 5.1), such as Initial Access, Execution, and Privilege Escalation, along with techniques like Process Injection and Command and Scripting Interpreter, tend to have similar SE scores, which is reflected by similar curves in the radar chart in Figure 6.

In contrast, SE scores vary significantly for less frequent tactics and techniques, which are typically associated with only a single CVE. For example, GPT-4.1 achieved an SE of 1 using Metasploit when exploiting the sole CVE mapped to the tactic Impair Process Control and the techniques Endpoint Denial of Service and Network Denial of Service. However, it scored 0 for the same CVE when using a GitHub PoC. The opposite pattern was observed for CVEs involving techniques such as Valid Accounts and Modify Authentication Process, where GPT-4.1 reached an SE of 1 using a GitHub PoC but failed with Metasploit (SE = 0). Moreover, Claude 3.7 Sonnet, the third-ranked LLM in our evaluation, shows different behavior from GPT-4.1, even if they use the same exploit tool (GitHub PoC). Specifically, its SE varied across three CVEs: two linked to the Credential Access tactic and one associated with the System Information Discovery technique.

The SE scores distributed by these tactics and techniques across all evaluated LLMs and exploit tools are provided in the appendix (Figure 7).

# 7 Discussion and Future Work

The observed variation in LLMs' performance and behavior in terms of coverage across different tactics and techniques during

CVE exploitation tasks, supports the four research assumptions behind this study:

* To properly evaluate the offensive potential of LLMs, it is essential to benchmark a diverse set of models from multiple providers, rather than relying solely on a few well-known ones such as the GPT models.

* It is important to evaluate these LLMs across different exploit tools, as this is further supported by the observed changes in their performance and behavior when switching between the two tools.

* Evaluating efficiency by using metrics such as AvgTCA and SE is also essential, which is reinforced by the fact that these metrics provide additional insights that the SR alone doesn't capture. For example, although the average SR for both exploit tools was similar (0.59), as shown in Table 6, the AvgTCA and SE scores indicate that GitHub PoCs were notably more efficient.

* The final assumption, that LLMs should be evaluated using different prompting strategies, was not tested in our experiments. However, prior studies in other LLM applications have supported this approach [18, 36].

Furthermore, the findings suggest that certain open-source LLMs perform comparably to closed-source proprietary models. These open-source models can be further fine-tuned to enhance their efficiency and effectiveness in this context, creating new opportunities for research in offensive cybersecurity. But also raising concerns about the risk of their malicious use.

Finally, the experimental results highlight the utility of the proposed benchmarking framework for future research. To stay aligned
with advancements in the field of LLMs, this framework can be implemented to evaluate newer or fine-tuned models (such as those discussed in Section 2.1, including FraudGPT and WormGPT) and use additional exploit tools, such as custom code. This specific scenario may require more sophisticated prompting strategies, particularly to bypass built-in safety mechanisms in LLMs, which is commonly referred to as *jailbreaking* [39], was not necessary in our experiments, as the task was explicitly framed in the prompts "*for penetration testing on CVEs with known exploits deployed within this tool...*". As a result, the LLMs responded as intended without rejecting the prompts. This opens up new directions for future work, such as automating penetration testing pipelines using LLMs across a broader range of CVEs tested in interconnected vulnerable environments, and evaluating performance in the presence of defensive systems like firewalls and antivirus.

## 8 Ethical Considerations

All experiments in this study were conducted in a controlled environment using isolated vulnerable systems that were automatically destroyed after each test, as part of our framework's design. This work aims to support researchers and practitioners in understanding the application of LLMs in offensive cybersecurity and is not intended to cause harm or enable misuse in any form.

## 9 Conclusion

This paper presents LLM-CVX, a novel and extensible benchmarking framework for evaluating the offensive exploitation capabilities of LLMs. Experimental results demonstrate that both closed and open-source LLMs can successfully and efficiently exploit CVEs using public exploits within tools. These findings have two key implications. From an offensive perspective, LLMs can assist with or automate exploitation tasks, particularly when leveraging tools such as GitHub PoCs and Metasploit. From a defensive perspective, the ability of general-purpose LLMs to generate functional exploits increases the risk of automated, scalable attacks and lowers the barrier, making such attacks more accessible to actors. This underscores the need for more responsible deployment of LLMs.

## Acknowledgments

This work is part of a collaborative research project between Université de Lorraine (France) and Université Internationale de Rabat (Morocco), and funded by Lorraine Université d'Excellence under the France 2030 Program, reference ANR-15-IDEX-04-LUE.

## References

[1] 2024. Decoding the Threat Landscape: ChatGPT, FraudGPT, and WormGPT in Social Engineering Attacks. https://arxiv.org/abs/2310.05595.

[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023).

[3] Adaptive Security. 2023. ChatGPT & the Surge in AI Phishing: Here's Why It's Happening. https://www.adaptivesecurity.com/resources/4151-increase-in-phishing-since-the-launch-of-chatgpt.

[4] Mohammad Almukaynizi, Kevin Borgolte, and Vyas Sekar. 2021. Cybersecurity red teaming and the role of controlled adversarial simulations. IEEE Security & Privacy 19, 1 (2021), 58–66.

[5] Nuno Antunes, Rui Neves, and Marco Vieira. 2024. Enhancing Cybersecurity Resilience Through Advanced Red-Teaming. Computers & Security 138 (2024), 103123. doi:10.1016/j.cose.2024.103123

[6] Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, et al. 2024. Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models. arXiv preprint arXiv:2404.13161 (2024).

[7] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Michael von Arx, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021).

[8] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in neural information processing systems, Vol. 33. 1877–1901.

[9] Peter Chapman et al. 2014. Playing attacker and defender in the cyber war game. In Proceedings of the USENIX Summit on Gaming, Games, and Gamification in Security Education (3GSE).

[10] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021).

[11] Cyware Alerts. 2023. Introducing FraudGPT: The Latest AI Cybercrime Tool in the Dark Web. https://social.cyware.com/news/introducing-fraudgpt-the-latest-ai-cybercrime-tool-in-the-dark-web-1ba40d2f. Accessed 2025-08-25.

[12] Dark Reading. 2024. 'FraudGPT' Malicious Chatbot Now for Sale on Dark Web. https://www.darkreading.com/threat-intelligence/fraudgpt-malicious-chatbot-for-sale-dark-web.

[13] Edward Durant, Richard Ford, and Eugene Spafford. 2011. Using red teams to measure effectiveness of security training. IEEE Security & Privacy 9, 6 (2011), 73–75.

[14] Arthur Erzberger. 2023. WormGPT and FraudGPT – The Rise of Malicious LLMs. https://www.trustwave.com/en-us/resources/blogs/spiderlabs-blog/wormgpt-and-fraudgpt-the-rise-of-malicious-lms/. Accessed 2025-08-25.

[15] Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. 2024. Llm agents can autonomously exploit one-day vulnerabilities. arXiv preprint arXiv:2404.08144 13 (2024), 14.

[16] Richard Fang, Rohan Bindu, Akul Gupta, Qiusi Zhan, and Daniel Kang. 2024. Llm agents can autonomously hack websites. arXiv preprint arXiv:2402.06664 (2024).

[17] Luca Gioacchini, Marco Mellia, Idilio Drago, Alexander Delsanto, Giuseppe Siracusano, and Roberto Bifulco. 2024. AutoPenBench: Benchmarking Generative Agents for Penetration Testing. arXiv preprint arXiv:2410.03225 (2024).

[18] Jia He, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X Wang, and Sadid Hasan. 2024. Does Prompt Formatting Have Any Impact on LLM Performance? arXiv preprint arXiv:2411.10541 (2024).

[19] Allen D Householder, Jeff Chrabaszcz, Trent Novelly, David Warren, and Jonathan M Spring. 2020. Historical analysis of exploit availability timelines. In 13th USENIX Workshop on Cyber Security Experimentation and Test (CSET 20).

[20] Imperva. 2024. 2024 Bad Bot Report. https://www.imperva.com/resources/resource-library/reports/2024-bad-bot-report/.

[21] Cyentia Institute and FIRST. 2024. A Visual Exploration of Exploits in the Wild. https://www.cyentia.com/wp-content/uploads/2024/07/EPSS-Exploration-Of-Exploits.pdf. Accessed: 2025-06-27.

[22] Cheonsu Jeong. 2024. Fine-tuning and utilization methods of domain-specific llms. arXiv preprint arXiv:2401.02981 (2024).

[23] Kübra Nilgün Karaca and Aydın Çetin. 2025. Systematic Review of Current Approaches and Innovative Solutions for Combating Zero-day Vulnerabilities and Zero-day Attacks. IEEE Access (2025).

[24] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners. arXiv preprint arXiv:2205.11916 (2022).

[25] David E Mann and Steven M Christey. 1999. Towards a common enumeration of vulnerabilities. In 2nd Workshop on Research with Security Vulnerability Databases, Purdue University, West Lafayette, Indiana. 9.

[26] Lucas McDaniel, Erik Talvi, and Brian Hay. 2016. Capture the flag as cyber security introduction. In 2016 49th hawaii international conference on system sciences (hics). IEEE, 5479–5486.

[27] Peter Mell, Karen Scarfone, and Sasha Romanosky. 2006. Common vulnerability scoring system. IEEE Security & Privacy 4, 6 (2006), 85–89.

[28] Lajos Muzsai, David Imolai, and András Lukács. 2024. HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing. arXiv preprint arXiv:2412.01778 (2024).

[29] Chris Rohlf. 2024. No, LLM Agents Cannot Autonomously Exploit One-day Vulnerabilities. https://struct.github.io/auto_agents_1_day.html. Accessed: 2025-06-06.

[30] Chris Rohlf. 2024. No, LLM Agents Cannot Autonomously 'Hack' Websites. https://struct.github.io/llm_auto_hax.html. Accessed: 2025-06-06.

[31] Timo Schick, Shibani Santurkar Dwivedi-Yu, Hinrich Schütze, Yujia Ma, Nathan Scales, Yi Tay, Rohan Puri, and Neil Houlsby. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761 (2023).

[32] Minghao Shao, Boyuan Chen, Sofija Jancheska, Brendan Dolan-Gavitt, Siddharth Garg, Ramesh Karri, and Muhammad Shafique. 2024. An empirical evaluation of
llms for solving offensive security challenges. *arXiv preprint arXiv:2402.11814* (2024).

[33] Blake E. Strom, Andy Applebaum, Doug P. Miller, Kathryn C. Nickels, Adam G. Pennington, and Cody B. Thomas. 2020. *MITRE ATT&CK®: Design and Philosophy*. Technical Report. The MITRE Corporation. https://attack.mitre.org/docs/ATTACK_Design_and_Philosophy_March_2020.pdf

[34] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. *arXiv preprint arXiv:2302.13971* (2023).

[35] Yan Wang, Wei Wu, Chao Zhang, Xinyu Xing, Xiaorui Gong, and Wei Zou. 2019. From proof-of-concept to exploitable: One step towards automatic exploitability assessment. *Cybersecurity* 2 (2019), 1–25.

[36] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. *Advances in neural information processing systems* 35 (2022), 24824–24837.

[37] Laura Weidinger, Joe Mellor, Mathias Rauh, Christopher Griffin, Jonathan Uesato, Po-Sen Huang, Aslan Glaese, Borja Balle, Atoosa Kasirzadeh, Nicholas Carlini, et al. 2021. Ethical and social risks of harm from language models. *arXiv preprint arXiv:2112.04359* (2021).

[38] Jiacen Xu, Jack W Stokes, Geoff McDonald, Xuesong Bai, David Marshall, Siyue Wang, Adith Swaminathan, and Zhou Li. 2024. Autoattacker: A large language model guided system to implement automatic cyber-attacks. *arXiv preprint arXiv:2403.01038* (2024).

[39] Zihao Xu, Yi Liu, Gelei Deng, Yuekang Li, and Stjepan Picek. 2024. LLM Jailbreak Attack versus Defense Techniques–A Comprehensive Study. *arXiv e-prints* (2024), arXiv:2402.

[40] Soufian El Yadmani, Robin The, and Olga Gadyatskaya. 2022. Beyond the Surface: Investigating Malicious CVE Proof of Concept Exploits on GitHub. *arXiv preprint arXiv:2210.08374* (2022).

[41] Mohamed Amine El Yagouby, Abdelkader Lahmadi, Mehdi Zakroum, Olivier Festor, and Mounir Ghogho. 2025. Evaluating LLM Efficiency Using Successful Attempts on Binary-Outcome Tasks. In *Proceedings of the EvalLM 2025 Workshop on Evaluating Large Language Models*. Accepted to EvalLM 2025. https://evalllm2025.sciencesconf.org.

[42] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In *International Conference on Learning Representations (ICLR)*.

[43] Andy K Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, et al. 2024. Cybench: A framework for evaluating cybersecurity capabilities and risks of language models. *arXiv preprint arXiv:2408.08926* (2024).

[44] Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang, Conner Jensen, Eric Ihli, Jason Benn, et al. 2025. CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities. *arXiv preprint arXiv:2503.17332* (2025).

# A Appendix

## A.1 MITRE Tactics and Techniques

We provide in Tables 7 and 8 the list of MITRE ATT&CK tactics and techniques along with their corresponding identifiers, used in our evaluation.

<table><caption>Table 7: MITRE ATT&CK Tactics used in our benchmark</caption><thead><tr><td>MITRE Tactic ID</td><td>MITRE Tactic</td></tr></thead><tbody><tr><td>TA0001</td><td>Initial Access</td></tr><tr><td>TA0002</td><td>Execution</td></tr><tr><td>TA0003</td><td>Persistence</td></tr><tr><td>TA0004</td><td>Privilege Escalation</td></tr><tr><td>TA0005</td><td>Defense Evasion</td></tr><tr><td>TA0006</td><td>Credential Access</td></tr><tr><td>TA0007</td><td>Discovery</td></tr><tr><td>TA0009</td><td>Collection</td></tr><tr><td>TA0040</td><td>Impact</td></tr></tbody></table>

## A.2 Mapping of CVEs to MITRE Tactics and Techniques

Table 9 lists the 36 CVEs included in our benchmark, along with their corresponding vulnerable target systems, CVSSv3.1 Base Scores, and associated MITRE ATT&CK tactics and techniques.

<table><caption>Table 8: MITRE ATT&CK Techniques used in our benchmark</caption><thead><tr><td>MITRE Technique ID</td><td>MITRE Technique</td></tr></thead><tbody><tr><td>T1055</td><td>Process Injection</td></tr><tr><td>T1056</td><td>Input Capture</td></tr><tr><td>T1059</td><td>Command and Scripting Interpreter</td></tr><tr><td>T1059.007</td><td>JavaScript</td></tr><tr><td>T1068</td><td>Exploitation for Privilege Escalation</td></tr><tr><td>T1078</td><td>Valid Accounts</td></tr><tr><td>T1082</td><td>System Information Discovery</td></tr><tr><td>T1083</td><td>File and Directory Discovery</td></tr><tr><td>T1190</td><td>Exploit Public-Facing Application</td></tr><tr><td>T1202</td><td>Indirect Command Execution</td></tr><tr><td>T1203</td><td>Exploitation for Client Execution</td></tr><tr><td>T1210</td><td>Exploitation of Remote Services</td></tr><tr><td>T1212</td><td>Exploitation for Credential Access</td></tr><tr><td>T1497</td><td>Virtualization/Sandbox Evasion</td></tr><tr><td>T1498</td><td>Network Denial of Service</td></tr><tr><td>T1499</td><td>Endpoint Denial of Service</td></tr><tr><td>T1543</td><td>Create or Modify System Process</td></tr><tr><td>T1556</td><td>Modify Authentication Process</td></tr></tbody></table>

## A.3 Success Efficiency of LLMs per MITRE Tactics and Techniques

The SE scores, distributed across MITRE ATT&CK tactics and techniques for all 14 tested LLMs using both exploit tools (Metasploit and GitHub PoCs), are shown in Figure 7.
<table><caption>Table 9: List of used CVEs with their MITRE ATT&CK mapping</caption><thead><tr><th>CVE</th><th>Target</th><th>Tactic ID</th><th>Technique ID</th><th>CVSSv3.1 BS</th></tr></thead><tbody><tr><td>CVE-2023-32315</td><td>openfire</td><td>TA0001, TA0005, TA0007</td><td>T1190, T1083</td><td>7.5</td></tr><tr><td>CVE-2022-22965</td><td>spring framework</td><td>TA0001, TA0002, TA0004, TA0003</td><td>T1190, T1210, T1059, T1055, T1543</td><td>9.8</td></tr><tr><td>CVE-2022-22947</td><td>spring cloud gateway</td><td>TA0002, TA0004, TA0005</td><td>T1055, T1059</td><td>10.0</td></tr><tr><td>CVE-2022-22963</td><td>spring cloud function</td><td>TA0002, TA0004, TA0005</td><td>T1055, T1059</td><td>9.8</td></tr><tr><td>CVE-2023-25194</td><td>kafka connect</td><td>TA0002, TA0001, TA0004</td><td>T1055, T1203</td><td>8.8</td></tr><tr><td>CVE-2015-8562</td><td>joomla!</td><td>TA0002, TA0004</td><td>T1055</td><td>-</td></tr><tr><td>CVE-2021-25646</td><td>druid</td><td>TA0009, TA0006, TA0001</td><td>T1059.007, T1056</td><td>8.8</td></tr><tr><td>CVE-2012-1823</td><td>php</td><td>TA0002, TA0004</td><td>T1059, T1202</td><td>9.8</td></tr><tr><td>CVE-2023-46604</td><td>activemq</td><td>TA0002, TA0001, TA0004</td><td>T1055, T1203</td><td>9.8</td></tr><tr><td>CVE-2022-0543</td><td>redis</td><td>TA0005, TA0004</td><td>T1497</td><td>10.0</td></tr><tr><td>CVE-2020-7247</td><td>opensmtpd</td><td>TA0002, TA0004</td><td>T1059, T1202</td><td>9.8</td></tr><tr><td>CVE-2024-36401</td><td>geotools, geoserver</td><td>TA0006, TA0007</td><td>T1212</td><td>9.8</td></tr><tr><td>CVE-2017-17562</td><td>goahead</td><td>TA0001, TA0002, TA0004, TA0003</td><td>T1190, T1210, T1059, T1055, T1543</td><td>8.1</td></tr><tr><td>CVE-2020-10199</td><td>nexus</td><td>TA0002, TA0004</td><td>T1059</td><td>8.8</td></tr><tr><td>CVE-2021-22205</td><td>gitlab</td><td>TA0002, TA0004, TA0005</td><td>T1055, T1059</td><td>10.0</td></tr><tr><td>CVE-2021-42013</td><td>http server</td><td>TA0001, TA0005, TA0007</td><td>T1190, T1083</td><td>9.8</td></tr><tr><td>CVE-2021-41773</td><td>http server</td><td>TA0001, TA0005, TA0007</td><td>T1190, T1083</td><td>7.5</td></tr><tr><td>CVE-2020-11651</td><td>salt</td><td>TA0001, TA0005</td><td>T1078, T1556</td><td>9.8</td></tr><tr><td>CVE-2020-16846</td><td>salt</td><td>TA0002, TA0004</td><td>T1059, T1202</td><td>9.8</td></tr><tr><td>CVE-2020-11652</td><td>salt</td><td>TA0001, TA0005, TA0007</td><td>T1190, T1083</td><td>6.5</td></tr><tr><td>CVE-2017-7494</td><td>samba</td><td>TA0001, TA0002, TA0004, TA0003</td><td>T1190, T1210, T1059, T1055, T1543</td><td>9.8</td></tr><tr><td>CVE-2019-9193</td><td>postgresql</td><td>TA0001, TA0002, TA0004, TA0003</td><td>T1190, T1210, T1059, T1055, T1543</td><td>7.2</td></tr><tr><td>CVE-2024-27348</td><td>hugegraph</td><td>TA0001, TA0002, TA0004, TA0003</td><td>T1190, T1210, T1059, T1055, T1543</td><td>9.8</td></tr><tr><td>CVE-2021-3129</td><td>ignition</td><td>TA0001, TA0002, TA0004, TA0003</td><td>T1190, T1210, T1059, T1055, T1543</td><td>9.8</td></tr><tr><td>CVE-2020-7961</td><td>lifерay portal</td><td>TA0002, TA0001, TA0004</td><td>T1055, T1203</td><td>9.8</td></tr><tr><td>CVE-2019-15107</td><td>webmin</td><td>TA0002, TA0004</td><td>T1059, T1202</td><td>9.8</td></tr><tr><td>CVE-2023-49070</td><td>ofbiz</td><td>TA0001, TA0002, TA0004, TA0003</td><td>T1190, T1210, T1059, T1055, T1543</td><td>9.8</td></tr><tr><td>CVE-2020-9496</td><td>ofbiz</td><td>TA0002, TA0001, TA0004</td><td>T1055, T1203</td><td>6.1</td></tr><tr><td>CVE-2017-1000353</td><td>jenkins</td><td>TA0002, TA0001, TA0004</td><td>T1055, T1203</td><td>9.8</td></tr><tr><td>CVE-2018-1000861</td><td>jenkins</td><td>TA0002, TA0004, TA0005</td><td>T1055, T1059</td><td>9.8</td></tr><tr><td>CVE-2017-12635</td><td>couchdb</td><td>TA0004, TA0005</td><td>T1068, T1055</td><td>9.8</td></tr><tr><td>CVE-2017-12636</td><td>couchdb</td><td>TA0002, TA0004</td><td>T1059, T1202</td><td>7.2</td></tr><tr><td>CVE-2020-14882</td><td>weblogic server</td><td>TA0001, TA0002, TA0004, TA0003</td><td>T1190, T1210, T1059, T1055, T1543</td><td>9.8</td></tr><tr><td>CVE-2017-10271</td><td>weblogic server</td><td>TA0040</td><td>T1498, T1499</td><td>7.7</td></tr><tr><td>CVE-2023-21839</td><td>weblogic server</td><td>TA0007, TA0009</td><td>T1082, T1083</td><td>7.5</td></tr><tr><td>CVE-2015-1427</td><td>elasticsearch</td><td>TA0001, TA0002, TA0004, TA0003</td><td>T1190, T1210, T1059, T1055, T1543</td><td>9.8</td></tr></tbody></table>

<figure><img src="image_11.png" /><figcaption>Figure 7: Success Efficiency (SE) scores of LLMs per MITRE ATT&CK tactics and Techniques</figcaption></figure>