DOI: 10.5281/zenodo.16366788
title: Explicit Vulnerability Generation with LLMs: An Investigation Beyond Adversarial Attacks

description: Vulnerable Code Generation with Open-weight LLMs

Overview

This repository contains the scripts and setup to evaluate the generation of vulnerable code by open-weight Large Language Models (LLMs) in response to various prompts. We use dynamic prompting (zero-shot) and reverse prompting (one-shot) techniques to guide the models (Qwen2, Mistral, and Gemma) to generate C code that contains specific vulnerabilities.

Accepted Paper https://arxiv.org/abs/2507.10054:

@misc{bosnak2025explicitvulnerabilitygenerationllms,
      title={Explicit Vulnerability Generation with LLMs: An Investigation Beyond Adversarial Attacks}, 
      author={Emir Bosnak and Sahand Moslemi and Mayasah Lami and Anil Koyuncu},
      year={2025},
      eprint={2507.10054},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2507.10054}, 
}


Repository Structure

.
├── analysis - Contains analysis and vulnerability detection scripts
│   ├── esbmc - ESBMC analysis tools and scripts  
│   ├── esbmc-reports - Raw ESBMC output logs
│   ├── full_analysis_dynamic_prompting.py - Dynamic prompting analysis script
│   ├── full_analysis_reverse_prompting.py - Reverse prompting analysis script  
│   └── vuln-analysis-summary - ESBMC analysis summaries
├── c-code - Extracted C source files from model outputs
│   ├── dynamic - C files from dynamic prompting
│   └── reverse - C files from reverse prompting
├── compact_horizontal_plot_with_backgrounds.png - Figure visualization
├── final-summary - Final analysis reports
│   ├── dynamic_prompting_full_analysis.txt - Dynamic prompting results
│   └── reverse_prompting_full_analysis.txt - Reverse prompting results
├── FormAI-v2 - FormAI-v2 dataset files
│   └── FormAI-v2.json - Metadata and vulnerability annotations
├── full_generation_visual.py - Figure 5 generation script
├── heatmap_correct_vuln.png - Heatmap visualization
├── heatmap.py - Figure 6 generation script
├── kaggle-notebooks - Jupyter notebooks for model inference
│   ├── gemma.ipynb - Gemma model notebook
│   ├── gemma_temp.ipynb - Gemma temperature experiment
│   ├── mistral.ipynb - Mistral model notebook
│   ├── mistral_temp.ipynb - Mistral temperature experiment
│   ├── qwen2.ipynb - Qwen2 model notebook
│   └── qwen2_temp.ipynb - Qwen2 temperature experiment
├── model-outputs - JSON files containing model generation results
│   ├── dynamic - Dynamic prompting model outputs
│   └── reverse - Reverse prompting model outputs
├── prompt-generation - Prompt generation scripts and files
│   ├── dynamic - Dynamic prompting generation
│   └── reverse - Reverse prompting generation
├── README.md - This file
├── requirements.txt - Python dependencies
└── sampled_code - Sampled vulnerable code by vulnerability type
    ├── buffer-overflow - Buffer overflow samples
    ├── dereference-failure - Dereference failure samples
    ├── integer-overflow - Integer overflow samples
    ├── out-of-bounds - Out-of-bounds access samples
    └── use-after-free - Use-after-free samples


Requirements

OS and Environment



The experiments were run on Kaggle Notebooks using default settings.

The analysis scripts were run on Ubuntu 24.04 TLS

Python 3.x environment with the following dependencies:



transformers

accelerate

einops

bitsandbytes




These dependencies are listed in the requirements.txt file.

Models



Qwen2-7B-Instruct

Mistral-7B-Instruct

Gemma-7B-Instruct


These models were loaded using the Hugging Face transformers library with quantization options for efficient inference.

GPU/Hardware



The experiment was conducted on Kaggle using their GPU P100 accelerator.


Additional Software



ESBMC (Extended Symbolic Model Checker) - latest version from GitHub

7zip for dataset extraction

Gemini API key (for reverse prompt generation)


Installation



Install the necessary Python dependencies:


pip install -r requirements.txt




Install 7zip:


sudo apt install 7zip




Install ESBMC:


Download and extract the latest esbmc-linux.zip from the ESBMC GitHub repo (github.com/esbmc/esbmc):

# Download the latest esbmc-linux.zip from ESBMC GitHub Releases
unzip esbmc-linux.zip -d esbmc
echo 'export PATH="$HOME/esbmc/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc


Verify ESBMC installation:

esbmc --version


Expected output: ESBMC version information



Set up the FormAI-v2 dataset:


git clone https://github.com/FormAI-Dataset/FormAI-dataset/
7z x FormAI-dataset/FormAI-v2-classification.7z.001
mkdir -p FormAI-v2
mv FormAI-v2.json FormAI-v2/




Basic usage test - Generate dynamic prompts:


sudo python3 prompt-generation/dynamic/dynamic_prompt_generation.py


Expected output: Saved 2040 prompts to /prompt-generation/dynamic/dynamic_prompts.json



Test ESBMC integration by extracting sample code:


python analysis/esbmc/extract_code.py --input path_to_model_output_file


This will create compilable C files in the ./c-code directory, confirming that the code extraction and compilation validation pipeline is working correctly.

Installation tested on: Ubuntu 24.04 TLS with Python 3.x, using Kaggle GPU P100 accelerator for model inference.

Dataset Setup

FormAI-v2 Dataset

This project builds on the FormAI-v2 dataset, which provides vulnerable C code samples generated by large language models. Make sure to place FormAI-v2.json in a directory named FormAI-v2/ in this project root. This file is required to sample real vulnerabilities and generate prompts.

To get FormAI-v2.json:



Install 7zip:


sudo apt install 7zip




Clone the repository:


git clone https://github.com/FormAI-Dataset/FormAI-dataset/




Extract the classification files:


7z x FormAI-dataset/FormAI-v2-classification.7z.001


This command will automatically handle both parts (.7z.001 and .7z.002) and extract the FormAI-v2.json file.



Move FormAI-v2.json to ./FormAI-v2/ folder


FormAI-v2.json contains metadata and vulnerability annotations for each file in the dataset.

Each entry in the FormAI-v2.json file includes:



filename: name of the C source file (also the model name in the file name).

vuln_line and vuln_column: the location of the vulnerability in the code.

violation: the violated ESBMC property (i.e., the vulnerability type).

snippet: a short code excerpt showing the vulnerable region.

cyclomatic_complexity: a complexity score used for bucketed analysis.


Experiment Commands

Dynamic Prompting (Zero-shot) - prompt-generation/dynamic



Generate the Prompts: To generate the dynamic prompts, run the following script:


sudo python3 prompt-generation/dynamic/dynamic_prompt_generation.py


Since sudo uses Python 2 by default, using sudo python ... might result in an error.

Expected stdout: Saved 2040 prompts to /prompt-generation/dynamic/dynamic_prompts.json



Run the Models on Prompts - ./kaggle-notebooks The models are run using the respective Jupyter notebooks for Qwen2, Mistral, and Gemma, located in the ./kaggle-notebooks directory. The generated prompts are provided as input to these notebooks by setting the variable prompt_path = "/kaggle/input/new-prompts/dynamic_prompts.json".


The outputs will be saved to results/<model_name>_results.json.

./model_outputs contains these JSON files.



Sample 100 Prompts (For Temperature Experiment) If you wish to run the temperature experiment with a sample of prompts, run:


python prompt-generation/dynamic/sample_prompts.py


Expected stdout: Saved 100 prompts to .../prompt-generation/dynamic/dynamic_prompts_temp_sample.json

Reverse Prompting (One-Shot) - sampled_code Generation - prompt-generation/reverse



Extract Vulnerable Code from FormAI-v2:


python prompt-generation/reverse/extract_vulnerable_code.py


Expected stdout: [done] Extracted 742252 vulnerable files into .../prompt-generation/reverse/formai_all_vulnerable_code

The script takes input from ./FormAI-v2/FormAI-v2.json. It filters valid, verified vulnerable code samples, categorizes them by vulnerability type and cyclomatic complexity, and saves the source code as .c files in ./prompt-generation/reverse/formai_all_vulnerable_code/<vuln_type>/<cc_bucket>/.



Get a Sample of Vulnerable Code:


python prompt-generation/reverse/get_sample_code.py


The script takes input from ./prompt-generation/reverse/formai_all_vulnerable_code/, randomly samples up to 5 .c files per subtype and cyclomatic complexity bucket, and saves them to ./prompt-generation/reverse/sampled_code/<vuln_type>/<subtype>/<cc_bucket>/. It organizes samples by vulnerability type, subtype, and CC bucket, resolving subtype-to-folder mappings where needed.



Generate Reverse Prompts Using Gemini API (or any other LLM): Add your Gemini API key to the generate_reverse_prompts.py file and run:


python prompt-generation/reverse/generate_reverse_prompts.py


You can also change the Gemini setup and call_llm(prompt) function with a commercial LLm of your choice. However, this could impact the final results.

ESBMC Setup - /analysis

Install ESBMC

Download and extract the latest esbmc-linux.zip from the ESBMC GitHub repo (github.com/esbmc/esbmc) and add the binary to your environment's path.

1.1 Download: Get the latest esbmc-linux.zip from the ESBMC GitHub Releases.

1.2 Extract:

unzip esbmc-linux.zip -d esbmc


1.3. Add esbmc/bin to PATH:

echo 'export PATH="$HOME/esbmc/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc


1.4. Verify:

esbmc --version


Run ESBMC Analysis



Run ESBMC Analysis After collecting model outputs, extract C code with:


python analysis/esbmc/extract_code.py --input path_to_model_output_file


Enter the model result filename (e.g., reverse/qwen2/qwen2_reverse_prompting_full.json):

Before running the script, ensure that the model outputs are placed in the following path format: ./model-outputs/<dynamic|reverse>/<Model>/<JSON File> (e.g., ./model-outputs/reverse/qwen2/qwen2_reverse_prompting_full.json).

Then, provide the relative path from ./model-outputs as the input to process the file (e.g., reverse/qwen2/qwen2_reverse_prompting_full.json).

The output will be stored in the ./c-code directory. The script saves the extracted .c source files and temporary .out binaries generated during compilation.

Files are saved under the following structure: ../../c-code/<mode>/<model>/

Example (based on user input reverse/qwen2/qwen2_reverse_prompting_full.json): ../../c-code/reverse/qwen2/

After attempting to compile each .c file, the script removes:



Any .c files that fail to compile

All temporary .out binary files generated during compilation


Only compilable .c files are retained in the output directory.



Run ESBMC on Extracted Code


python analysis/esbmc/esbmc_analysis.py --input path_to_extracted_code


Before running this script, ensure that you have already generated the extracted .c files by completing the previous step.

This script analyzes the extracted C files using ESBMC. It expects the same relative model result path provided in the previous step (e.g., reverse/qwen2/qwen2_reverse_prompting_full.json) as input.

It performs the following actions:

Locates the corresponding .c files under ./c-code/<dynamic|reverse>/<Model>/, based on the input path.

Runs ESBMC on each .c file with a 15-second timeout, using verification flags such as --incremental-bmc and --overflow-check.

Writes raw ESBMC output logs to ./analysis/esbmc-reports/<mode>/, one .txt file per C file.

Generates two JSON summaries in ./analysis/vuln-analysis-summary/<mode>/:

A detailed summary with ESBMC results, matched metadata (e.g., vulnerability type, prompt), and violated properties if any.

An executive summary with counts of each result type and, if applicable, a breakdown by temperature.

Timeouts, parsing errors, and verification failures are identified and reported. Only .c files in the extracted code directory are processed.

Full Analysis

Once the C code is extracted and analyzed, run the following scripts for full analysis:

cd analysis
python full_analysis_dynamic_prompting.py
python full_analysis_reverse_prompting.py


These will generate detailed analysis summaries, which are stored in the ./final-summary directory.

Figures & Tables

full_generation_visual.py and heatmap.py use hardcoded data from ./final-summary and generate figures 5 and 6 in the paper respectively.

The results in tables 1 to 4 are manually taken from ./final-summary.

./final-summary includes two files, ./final-summary/dynamic_prompting_full_analysis.txt and ./final-summary/reverse_prompting_full_analysis.txt. Each file is organized by model and research question.



The RQ1: Vulnerable‐code and correct‐vuln counts part in ./final-summary/dynamic_prompting_full_analysis.txt and ./final-summary/reverse_prompting_full_analysis.txt contains results for Table-1

The RQ2: By intent (absolute counts) parts contains results for Table-2.

The RQ3: By form (absolute counts) parts contains results for Table-3.

The RQ4: Incorrect Vulnerability Generation Patterns parts contains results for Table-4.


Notes



The "total_prompts": 2040, "code_generated": 2034, "compilable_code": 1711 attributes in the executive summary need to be added manually by checking the results from ./analysis/esbmc/extract_code.py (this part was not automated).

For reverse prompt generation, Gemini API key is required. You can use any other LLM API, but be aware that results may differ significantly due to different model architectures and training data.


Outputs and Results

All intermediate and final results—including model generations, C code files, ESBMC reports, and summaries—are included in the replication release v1.0. These outputs are required to reproduce the tables and findings in the paper and are not tracked in the main branch due to their size.

Final Remarks

This repository provides a comprehensive framework for evaluating the security risks associated with the use of LLMs in generating code. The techniques can be adapted to other tasks involving code generation, vulnerability analysis, and prompt engineering for LLMs.