DOI: 10.5281/zenodo.18002402
title: Replication Package for the Paper: "An Insight into Security Code Review with LLMs: Capabilities, Obstacles, and Influential Factors".

description: This is the replication package for the paper: "An Insight into Security Code Review with LLMs: Capabilities, Obstacles, and Influential Factors".

The replication package is organized into three folders:

1. RQ1 Performance of LLMs

- Five prompt templates.pdfThis PDF demonstrates the detailed structures of the five prompt templates designed in Section 2.3.2 of our paper.

-  the cot-guardrail prompt template.pdfThis PDF presents the detailed structure of the sixth prompt template designed in Section 2.3.2 of our paper, which is the guardrail version of the CoT prompt.

- source code of the Python and C/C++ datasetsThis folder contains the source code of the Python and C/C++ datasets, used to construct prompts and apply the baseline tools for static analysis.

- prompts for the Python and C/C++ datasetsThis folder contains the prompts constructed from the source code of the Python and C/C++ datasets based on the six prompt templates.

- responses of LLMs and baselinesThis folder contains the responses generated by LLMs for each prompt and the analysis results of baseline tools. For CodeQL, you need to upload results.sarif to GitHub (https://docs.github.com/en/code-security/code-scanning/integrating-with-code-scanning/uploading-a-sarif-file-to-github) to view the analysis results. For SonarQube, you need to import the export file into an Enterprise Edition or higher instance of the same version (v10.5 in our work) and similar configuration (default configuration in our work) to view the analysis results. 

- entropy_calculation.pyThis Python script calculates the average entropy of each llm-prompt combination to measure the consistency of LLM responses in three repetitive experiments.

- Data Labelling for the C/C++ Dataset.xlsx- Data Labelling for the Python Dataset.xlsxThe two Microsoft (MS) files contain the labeling results for LLMs and baselines in the C/C++ and Python datasets, including the category of each response generated by LLM for each prompt, as well as the category of each analysis result generated by baseline for each code file.  The four categories(i.e., Instrumental, Helpful, Misleading and Uncertain) are defined in Section 2.3.3 of our paper as the labelling criteria.

How to Read the MS Excel files:Both MS Excel files contain 5 sheets. The first sheet ('all_c++_data' or 'all_python_data') includes the information of all data in each dataset. The sheets 'first round', 'second round' and 'third round' represent the labelling results for LLMs under five prompts in three repetitive experiments. The sheet 'Baselines' include the labelling results for baseline tools.




Column
Description


File ID
the identifier of each code file in our dataset.


Security Defect
the security defect(s) that the code file contains.


Project  
the source project of the code file.


Suffix
the suffix of the code file.




 

2. RQ2 Quality Problem in Responses

- DeepSeek-R1



data_analysis_first_round



c.mx24

python.mx24



data_analysis_second_round



c.mx24

python.mx24



data_analysis_third_round



c.mx24

python.mx24




- GPT-4



data_analysis_first_round.mx24

data_analysis_second_round.mx24

data_analysis_third_round.mx24


This directory contains two sets of MAXQDA project files. They correspond to the data extraction results on quality problems in responses generated by the two best-performing LLMs: DeepSeek-R1 and GPT-4, under their optimal prompts across three repetitive experiments. We split the data extraction results of Deepseek-R1 in each round of experiments into two mx24 files by programming language (Python -> python.mx24, C/C++ -> c.mx24), to prevent a single file from being too large. The MAXQDA project file can be opened by MAXQDA 2024 or higher versions, which are available at https://www.maxqda.com/ for download. You may also use the free 14 days trial version of MAXQDA, which is available at https://www.maxqda.com/trial for download.

 

3. RQ3 Factor influencing LLMsThis folder contains two sub-folders:

- Step 1 - correlation analysisFiles in this subfolder are for conducting correlation analysis for explanatory variables through a Python script.

- Step 2 - redundancy analysis and model fittingFiles in this subfolder are for conducting redundancy analysis, allocation of degree of freedoms, model fitting and evaluation through an R script. Detailed instructions for running the R script can be found in readme.md in this subfolder.