To Err is Machine: Vulnerability Detection Challenges LLM Reasoning

## Abstract

Abstract Large Language Models (LLMs) have demonstrated great potential for code generation and other software engineering tasks.
Vulnerability detection is of crucial importance to maintaining the security, integrity, and trustworthiness of software systems. Precise vulnerability detection requires reasoning about the code, making it a good case study for exploring the limits of LLMs’ reasoning capabilities. Although recent work has applied LLMs to vulnerability detection using generic prompting techniques, their full capabilities for this task and the types of errors they make when explaining identified vulnerabilities remain unclear. In this paper, we surveyed eleven LLMs that are state-of-the-art in code generation and commonly used as coding assistants, and evaluated their capabilities for vulnerability detection.
We systematically searched for the best-performing prompts, incorporating techniques such as in-context learning and chain-of-thought, and proposed three of our own prompting methods.
Our results show that while our prompting methods improved the models’ performance, LLMs generally struggled with vulnerability detection. They reported 0.5-0.63 Balanced Accuracy and failed to distinguish between buggy and fixed versions of programs in 76% of cases on average.
By comprehensively analyzing and categorizing 287 instances of model reasoning, we found that 57% of LLM responses contained errors, and the models frequently predicted incorrect locations of buggy code and misidentified bug types.
LLMs only correctly localized 6 out of 27 bugs in DbgBench, and these 6 bugs were predicted correctly by 70-100% of human participants.
These findings suggest that despite their potential for other tasks , LLMs may fail to properly comprehend critical code structures and security-related concepts.
Our study highlights the need for further research to improve the vulnerability detection capabilities of LLMs.
Our data and code are available at https://figshare.com/s/78fe02e56e09ec49300b .

## I Introduction

Large Language Models (LLMs) have demonstrated impressive capabilities across various software engineering tasks, such as code generation [], test generation [], and code summarization []. Their versatile capabilities, along with their novel ability to explain their predictions in plain-text, make LLMs worthwhile subjects of investigation for detecting security vulnerabilities.
In static analysis and human inspection, vulnerability detection requires code understanding and reasoning. Pattern-matching on code structures is insufficient to produce precise analyses [, ], especially for real-world code. For example, to precisely detect a buffer overflow, we cannot only scan for strcpy or malloc statements. We need to identify the statements that update the strings and buffers, reason about the lengths of the strings after the changes at these statements, and also understand the bounds-check code to judge whether the protection is sufficient.

LLMs have shown limited ability for complex reasoning [], and a recent survey calls for more challenging tasks to evaluate their reasoning abilities []; this calls into question whether LLMs possess the capabilities to perform effective vulnerability detection. This capability has profound implications for the potential applications of LLMs in other tasks which require understanding and reasoning about semantics of code, e.g., fault localization, program repairs, debugging and code generation for non-trival tasks. Studying LLMs for vulnerability detection also shed lights for the industry practitioners on whether LLMs can be considered as their software assurance tools.

Several studies have shown that LLMs struggle to attain high performance on vulnerability detection [, , , ].
LLMs generally underperformed with basic prompts [], which motivated the use of in-context examples [], chain-of-thoughts [] and static analysis [].
Prior studies primarily evaluate vulnerability detection performance. The very recent work (to the best of our knowledge, these papers are still in arXiv but not yet published) highlight broad errors in the models’ vulnerability detection responses [, , , ], but do not comprehensively investigate the LLM errors from the perspective of detecting vulnerabilities.

Our studies, which are concurrently developed with these works, aim for (1) comprehensiveness — we surveyed a wider variety of models and prompts in different scenarios of vulnerability detection, and analyzed more LLM responses with multiple raters, compared to prior work; and
(2) usefulness — beyond evaluating “Yes/No” predictions, we studied whether LLMs can correctly report the types, locations and causes of a vulnerability, as done in industry-standard static analysis-based detectors [, , ], and point out capabilities and code structures which LLMs struggled with, highlighting future research directions.
In that regard, we also compared the capabilities of LLMs with human performance for understanding vulnerabilities.
Specifically, we designed the following four questions to guide our research:

- •
RQ1: What are the most successful and least successful prompt designs for LLM-based vulnerability detection?
- •
RQ2: How well do state-of-the-art LLMs perform on vulnerability detection?
- •
RQ3: What kinds of errors do LLMs make when explaining the vulnerabilities?
- •
RQ4: How do LLMs compare to human developers for localizing vulnerabilities?

To thoroughly investigate these RQs, we evaluated 11 SOTA LLMs using five distinct prompt templates, including developing three novel techniques for generating prompts. We systematically searched among the prompts to draw out the best performance from each model (Section III-A). We investigated their empirical performance and their capability to distinguish buggy code from the corresponding fixed versions (Section III-B).
We rigorously analyzed 287 LLM responses explaining their vulnerability predictions and categorized them into 4 comprehensive error categories specifically tailored for vulnerability detection, revealing the ways in which the LLMs frequently failed and which code structures the models struggled with, as well as the effects of such errors on the models’ explanation quality (Section III-C).
We evaluated the models’ fault localization abilities on DbgBench, comparing their performance and difficulties with those of professional human software engineers to understand their similarities and differences (Section III-D).

Our results show the following:
most models performed best with Basic and In-Context prompting. Our proposed CoT-SA and Contrastive Pairs, used with In-Context prompting, were beneficial for some models.
LLMs performed within 0.5-0.63 Balanced Accuracy, close to a random-guess baseline, and failed to distinguish buggy and fixed versions of code in 76% of cases on average.
57% of LLM responses contained errors in Code Understanding, Hallucination, Logic, and/or Commonsense Knowledge, and among others, LLMs especially struggled with correctly identifying bounds/null checks. Furthermore, LLMs frequently predicted erroneous location, type, or root cause of vulnerabilities, in spite of the correct binary label.
LLMs correctly located 6/27 bugs in DbgBench [], all of which were correctly diagnosed by at least one human. GPT-3 performed the best among all the models and correctly located 4/27 bugs.

In summary, we make the following contributions:

- •
We designed 3 new prompt templates for vulnerability detection, integrating information from bug-fixing patches, CVE descriptions, and static analyzers.
- •
We comprehensively evaluated the vulnerability detection capabilities of state-of-the-art LLMs, including testing them on pairs of buggy/fixed code.
- •
We analyzed 287 LLM responses to break down the errors which LLMs made, and provide insights and a dataset for future research on vulnerability detection using LLMs.
- •
We evaluated LLMs’ debugging abilities in comparison with humans on the DbgBench vulnerability benchmark.

## II Study Setup

### II-A Dataset

We evaluated the models’ vulnerability detection capabilities mainly using the SVEN dataset released by He and Vechev []. This dataset was collected from CVE records reporting real-world vulnerabilities from the CrossVul [], BigVul [], and VUDENC [] datasets.

SVEN is made of the source code of functions that were changed in vulnerability-fixing commits, where the version of the code before the commit is considered vulnerable and the version after the commit is considered non-vulnerable.
The authors curated the dataset by manually inspecting the commits based on their domain expertise to ensure that the data includes true security vulnerabilities/fixes and excludes data quality issues, including spurious commit message matches, tangled commits, and project-specific APIs.
Leveraging these labels, we focus on the binary classification setting: given a prompt containing a function’s source code, models predict whether the function belongs in the positive (vulnerable) or negative (non-vulnerable) class.

The entire SVEN dataset comprises 846 C/C++ functions. We used 100 functions (50 pairs of a vulnerable function and its corresponding fixed version) to perform the study. This is because (1) many real-world functions are very long and do not fit the context window of the LLMs under study; (2) running LLM queries is very costly. It takes an average of 2 GPU-minutes per query, totaling 3.4 GPU-hours for 100 functions for each prompt method per model, and about $180 of OpenAI credits in total. We studied a total of
11 models and for each model, we used 5 prompt methods
(§ III-A1). In addition, the models’ responses tended to vary when sampled multiple times. To address this variance, we ran the model inference 3 times with different random seeds and report the aggregated results.

We also used the Big-Vul [] and D2A [] datasets to source in-context examples and extra metadata to generate our prompts (see Section III-A).
We used DbgBench [] to compare the performances of LLMs and human for vulnerability detection (see Section III-D).

### II-B Models

We collected a list of large language models from two benchmark surveys [, ] as well as the highest-performing models from PapersWithCode []. We added several notable open-source models (e.g. StarCoder []) and excluded the models for which inference was not available via API (e.g. Bard []), not trained on source code (e.g. Pythia []), were trained only on distillations from ChatGPT (e.g. Vicuna []), or were superseded by later iterations on the same project (e.g. LlaMA 1 []).
In Table I, we list 11 models used in our study, including the number of parameters (if it is an open source model), the length of the context window, i.e., the maximum number of input tokens, and HumanEval $Pass@1$ score [], a metric that can somewhat reflect the models’ capabilities in generating source code.

### II-C Implementation

We wrote scripts to automatically query the models and parse the model output. For the open-source models, we used huggingface [] local text generation APIs and wrote a custom prompting framework. For GPT-3.5, GPT-4, and Gemini, we used the APIs hosted by the corresponding organizations.
Table II shows the text generation parameters we used for the models.
We selected values of the models’ hyperparameters intended to make their responses more deterministic, without sampling greedily (which can substantially reduce the performance).

**TABLE I: 11 models we studied, HumanEval $Pass@1$ is reported by PapersWithCode [] or the source papers.**
| Model | Parameters | Context Length | HumanEval |
| --- | --- | --- | --- |
| GPT-4 [] | - | 128k | 86.6 |
| Gemini 1.0 Pro [] | - | 32k | 67.7 |
| WizardCoder [] | 15B | 2k | 57.3 |
| Code LLAMA [] | 34B | 16k$\sim$100k | 48.8 |
| GPT-3.5 [] | - | 4k | 48.1 |
| Mixtral-MoE [] | 45B | 8k$\sim$128k | 40.2 |
| Mistral [] | 7B | 8k$\sim$128k | 30.5 |
| StarCoder [] | 15B | 8k | 30.4 |
| LLAMA 2 [] | 70B | 4k | 29.9 |
| StarChat-$\beta$ [] | 15B | 8k | - |
| MagiCoder [] | 7B | 16k$\sim$100k | - |

**TABLE II: Text generation parameters we used.**
| Parameter | HuggingFace | OpenAI | Google |
| --- | --- | --- | --- |
| Top-$p$ | 0.9 | 1.0 | 1.0 |
| Temperature | 0.1 | 0.1 | 0.1 |
| Max. tokens generated | 512 | 512 | 512 |

The models generally replied with plain-text sentences. We developed a regular expression-based method to extract categorical yes/no responses; in cases where the models had no categorical response, we considered the prediction incorrect.
In order to ensure the correctness of our extraction method, we selected 10% of the responses for manual observation and found that our extraction procedure had an error rate of less than 0.5%.
For future reuse, we open-source our code and document further implementation details in our data package [].

### II-D Metrics

We evaluated the models using the following four metrics: Accuracy, the proportion of correct predictions on all examples, Precision, the proportion of correct predictions out of the examples on which the model predicted positive (vulnerable), Recall, the proportion of correct predictions out of the examples which are labeled positive (vulnerable), and Balanced Accuracy, the mean of per-class accuracy for both positive and negative classes.

$$ $\text{BalancedAccuracy}=\frac{\frac{correct_{negative}}{examples_{negative}}+ \frac{correct_{positive}}{examples_{positive}}}{2}$ $$

We observed that LLMs often produce predictions that are highly imbalanced, such as labeling every example as “vulnerable” (see Section III-B), and that conventional metrics such as F1 and MCC tend to favor these severely-imbalanced classifiers (see our data package for details []).
We argue that an imbalanced classifier that pathologically predicts only one class is less desirable than one that demonstrates some ability to differentiate between classes. In order to reflect this, we primarily used Balanced Accuracy [] to compare model performance.

## III Study Designs and Results of RQs

Figure: Figure 1: A comparison of different prompting methods for each model on 50 buggy and fixed pairs (100 examples). Bar height shows the mean of three random seeds, with error bars indicating one standard deviation.
Refer to caption: x1.png

Figure: Figure 2: A comparison of different prompting methods for each model on 50 unpaired examples (unrelated in terms of bug fixes). Bar height shows the mean of three random seeds, with error bars indicating one standard deviation.
Refer to caption: x2.png

### III-A RQ1: What are the most successful and least successful prompt designs for LLM-based vulnerability detection?

#### III-A 1 Prompting methods

We used three existing prompting methods [], including
Basic (zero-shot) prompting, In-context ($n$-shot) prompting, and In-context prompting based on embedding similarity.
We also designed three new prompting methods, namely In-context prompting based on contrastive pair, Chain-of-Thought from CVE descriptions, and Chain-of-Thought from static analysis.

We first designed a system prompt to set the context: “I want you to act as a vulnerability detection system”, and we explored a variety of natural language descriptions:
(1) Basic query: “Is the following function buggy? Please answer Yes or No.” (We also tried “Is the following function vulnerable?”; however, our pilot study shows that it did not perform as well.)
(2) CWE list: This prompt starts with “Does the following function contain one of the following bug types?”, followed by a fixed list of bug types, e.g., “CWE-190: Integer Overflow”;
(3) Q/A: Begin the query with “Question:” and begin the model’s response with “Answer:”. This conditions the model to respond in a question-answering mode.

Here, we provide examples of inputs and responses for in-context learning [].
The in-context examples condition the model to reply in the same format as the example responses. The selection of in-context examples can impact the performance. We studied three settings: (1) randomly selected examples, (2) the examples that had similar embeddings to the query example, and (3) the examples from contrastive pairs (see below for details).

Choose the in-context examples which are most similar to the code in the query, where “similar” is approximated by the vector proximity between code embeddings.
Examples which are closer in embedding space can be more relevant to the query code [].
We surveyed several embedding models [, , ] and chose CodeBERT [] because it
performed best. We searched the SVEN dataset (the 100 examples we used) for similar examples.

We used contrasting pairs by providing the vulnerable version of the code (before the bug-fixing commit) and the fixed version (after the commit) as in-context examples in the same prompt. Since these two versions of the source code differ primarily in the portion related to the bug-fix, we hoped that this prompt template would highlight the cause of the bug and instruct the model to learn that the small differences in code can lead to different labels.

We designed “chain-of-thought” prompts by providing intermediate reasoning steps which lead to the answer, inspired by Wei et al. []. We use in-context examples from the Big-Vul dataset [], which includes the CVE bug reports. For vulnerable examples, we used the default in-context query and provide the chain-of-thought response. To produce such response, we adapt the descriptions in these bug reports to describe how the bug manifests.
For example, CVE-2017-9211 [] describes the vulnerability, including the symptoms, attack surface, and variable involved:

The crypto_skcipher_init_tfm function in crypto/skcipher.c in the Linux kernel through 4.11.2 relies on a setkey function that lacks a key-size check, which allows local users to cause a denial of service (NULL pointer dereference) via a crafted application.

We use this description as the CoT response and append “Therefore, the example is buggy” to complete the response.
For non-vulnerable examples, we provide the default in-context example query/response.

We also used the output buggy paths reported by static analysis tools to prepare the chains of thought prompt. The buggy path consists of a list of statements that can lead to the bug. We use in-context examples from the D2A dataset [], which lists buggy paths from the Infer static analyzer [] for several open-source C++ projects.
We convert the buggy paths to natural language descriptions and use them as the response. This is an example COT response for a buffer overflow vulnerability:

1. A buffer buf of size 10 is allocated at line 1.

2. An index i is initialized to a value in the range [0, 100] at line 2.

3. The index i is used to access buf at line 3.
This may exceed the bounds of buf.

We append “Therefore, the example is buggy” to complete the response.
For non-vulnerable examples, we provide the default response.

Considering the numerous ways the above prompting techniques can be combined, we systematically searched for high-performing prompts for each model.
For the basic prompting method, we evaluated four “modes” —
(1) basic query, (2) Q/A, (3) CWE list, and (4) both Q/A and CWE list — and designated the best-performing mode as the Basic prompting approach.

For the in-context prompting methods, we evaluated random selection (IC-Random), embedding similarity search (IC-Embedding), chain-of-thought from CVE descriptions (CoT-CVE), and chain-of-thought from static analysis (CoT-SA).
We did a pilot study (§ V) and found that 6 shots performed the best for most models and was the maximum number of examples that fit within most models’ context sizes. Thus, we used 6 shots for all in-context prompts.
For each approach, we used two configurations: (1) 6 randomly-selected examples as well as (2) 3 contrastive pairs. We then selected the best outcome for each model and prompting method.
We list examples of the prompts and more details in our data package [].

#### III-A 2 RQ1 Results

Figure 1 shows the performance of all prompting methods over 3 random seeds, compared to a hypothetical random-guess baseline. The x-axis shows the models, and the y-axis reported the performance using the metric of Balanced Accuracy. Our results show that the Basic and IC-Random prompting methods achieve the best. As summarized on the pie chart on the right in Figure 1, the two methods (the blue and the orange colored parts) each ranked the best for 4 out of 11 models.
We designed different approaches to select examples used for in-context learning; our results show no single approach fits all models.
The IC-Embedding prompting ranked the best for 2 models and CoT-SA ranked the best for 1.

Q/A and CWE-list substantially improved the performance of Basic queries for 8 out of 11 models.
Additionally, IC-Embedding and CoT-SA techniques were beneficial for some models; IC-Embedding slightly outperformed IC-Random on LLAMA 2 and GPT-4, while CoT-SA substantially outperformed other approaches on average on MagiCoder.
Contrastive Pairs performed better than non-contrastive settings in 45% of cases.
Surprisingly, CoT-CVE did not perform very well, with CoT-CVE performing worse than the random baseline on 6 out of 11 models.
Our results indicate that additional bug information can be useful but not always.
This may be due to irrelevant context [] (e.g. the bug report contains information irrelevant to code like versions of programs) and is corroborated by Sun et al. [].
Future work could explore alternative information such as bug reports and bug-fixing commits and different prompting methods that use the bug information.

We evaluated the models on a dataset containing pairs of vulnerable and fixed programs.
This represents a use-case such as using LLMs to detect vulnerabilities, and then checking again for the vulnerability after applying a bug-fixing patch.
While this aspect is shared with other benchmarks based on bug reports, like Big-Vul [] and CVEFixes [], it may not represent all settings of vulnerablity detection.
To represent an alternative scenario of scanning new procedures for vulnerabilities, we also plotted results on a random sample of 25 vulnerable and 25 non-vulnerable functions from our dataset that were unrelated in terms of bug fixes.
Figure 2 shows that Basic, IC-Random and IC-Embedding were still the best prompting methods for the models, improving 5, 3, and 2 models respectively.
On these unpaired data, Contrastive Pairs performed better than non-contrastive settings 73% of the time.
The varying results across settings indicate that different scenarios may benefit from different prompts.

Summary for RQ1:
Across two vulnerability detection settings,
Basic or IC-Random prompts performed the best for 7-8 models.
Integrating Q/A and CWE-list were useful for improving model performance.
IC-Embedding and CoT-SA performed best for 3-4 models. Contrastive Pairs were useful for predicting unpaired data.

Figure: (a) Performance on the paired dataset.
Refer to caption: x3.png

### III-B RQ2: How well do state-of-the-art LLMs perform on vulnerability detection?

To explore the capabilities of LLMs for vulnerability detection, we compared the performance of the best prompting method for each model over 3 random seeds, shown in Figure 2(a). The best-performing model, StarChat-$\beta$, reported 0.56 Balanced Accuracy, showing that the current SOTA LLMs were not able to perform well for vulnerability detection. The performance of the models ranged from 0.56 to 0.50, close to that of random guessing. We observe that the model face challenges of distinguishing non-vulnerable and vulnerable versions, and 7 out of 11 models made 75% or more predictions on the same class. Some models are severely biased towards one label, e.g., Mistral mostly predicts non-vulnerable, with only three exceptions.

Compared to the HumanEval performance showed in Table I,
we see that high code generation performance did not imply high vulnerability detection performance.
Despite their high code generation performance measured on HumanEval, WizardCoder, Code LLAMA, and Gemini performed poorly on vulnerability detection; while StarChat-$\beta$, StarCoder, and MagiCoder performed relatively better. These results indicate that effective vulnerability detection may require models to possess skills beyond code generation an it’s not that the bigger models always perform better (StarChat-$\beta$, StarCoder, and MagiCoder only contain 7B parameters as shown in Table I).
GPT-4 and Mixtral-MoE maintained similar rankings in both tasks, displaying versatility across different contexts.

Table III presents a further analysis on the models’ capabilities of distinguishing pairs of vulnerable and nonvulnerable code. In the table, under Column Can’t Distinguish, we show that on average, 76% of pairs cannot be distinguished. Under Both Correct and Both Wrong columns, our results indicate that the models can predict both versions correctly in some instances (16% of pairs), but there are also cases (8% of pairs) where the models can predict both versions incorrectly.

Similar to the scenario in Figure 2, Figure 2(b) plots model performance on 50 unpaired examples. We see that performance improved slightly, but still ranged close to the random-guess baseline – from 0.5 to 0.63 Balanced Accuracy. The models ranked differently for the two settings of vulnerability detection.
In Figure 2(b), the unpaired setting, GPT-4 performed the best.

**TABLE III: Further analysis on 50 pairs of vulnerable and non-vulnerable examples. Cell values display the number and percentage of pairs in each category**
| Model | Can’t Distinguish | Both Correct | Both Wrong |
| --- | --- | --- | --- |
| StarChat-Beta | 24 (48%) | 18 (36%) | 8 (16%) |
| GPT-4 | 36 (72%) | 9 (18%) | 5 (10%) |
| StarCoder | 35 (70%) | 10 (20%) | 5 (10%) |
| Mixtral-MoE | 36 (72%) | 10 (20%) | 4   (8%) |
| MagiCoder | 42 (84%) | 6 (12%) | 2   (4%) |
| GPT-3.5 | 40 (80%) | 7 (14%) | 3   (6%) |
| LLAMA 2 | 41 (82%) | 6 (12%) | 3   (6%) |
| WizardCoder | 21 (42%) | 17 (34%) | 12 (24%) |
| Code LLAMA | 44 (88%) | 5 (10%) | 1   (2%) |
| Mistral | 47 (94%) | 3   (6%) | 0   (0%) |
| Gemini | 49 (98%) | 1   (2%) | 0   (0%) |
| Average | 38 (76%) | 8 (16%) | 4   (8%) |

Summary for RQ2:
The SOTA LLMs performed poorly at vulnerability detection, with Balanced Accuracy ranging from 0.5 to 0.63.
All models frequently failed to distinguish the buggy and fixed versions of programs — in 76% of cases on average.

### III-C RQ3: What types of errors do LLMs make when explaining vulnerabilities?

#### III-C 1 Study setup

For a vulnerability detection tool to be useful, we hope it can present further vulnerability information such as its cause, types and locations. We prompted the LLMs to explain all predictions that they identified as vulnerable and analyzed their responses.
The prompt consists of three parts:
(1) the original prompt described in Section III-A, applying the best-case prompt for each model noted in Figure 2(a),
(2) the model’s original prediction, e.g., “Yes/No” for basic prompts or “Yes, the vulnerability is CWE-XXX: Description” for prompts with CWE information, and
(3) “Please explain the reasoning for this answer”. We manually inspected the results and found that the models generally make four types of errors:

##### Code understanding errors

LLMs produce incorrect assertions about programming language semantics or code structures, including inferences about the effects of statements.
For instance, sometimes an LLM incorrectly claims that a pointer is accessed without being checked, when the code contains the check; sometimes an LLM claims an integer operation will overflow when, considering the size of the integer and possible values of the operation implied by the prior code context, an overflow is impossible.

##### Common knowledge errors

LLMs make incorrect assumptions about the applications, environment, or code conventions.
For example, an LLM may claim that if a variable N_FDC is negative, an error will occur. However, a developer could reasonably interpret the all-capitalized N_FDC as a configuration variable representing the number of a certain resource (floppy-disk controllers), and assume it to be a non-negative integer by definition.

##### Logic errors

LLMs produce statements that are logically contradictory or make incorrect logical implications when reaching the conclusions of vulnerability.
For example, sometimes an LLM correctly claims that a variable x is checked, then later contradicts itself by stating, “since x is not checked, it will overflow…”.

##### Hallucination/Memorization/Repetition errors

LLMs produce hallucination (stating information that sounds correct but is not present in the prompt, such as pointing out a potential integer overflow in an arithmetic operation that doesn’t appear in the code), memorization (memorizing information from the training data),
and repetition (infinitely repeating certain text sequences, for instance: “1. The function is buggy because it is not checking for a null pointer. 2. The function is buggy because it is not checking for a null pointer. 3…”).

Figure: Figure 4: Error categories observed in responses from all LLMs. Bar width shows the number of errors observed in each category. Bar labels show the percentage of responses containing each category of error. Responses can contain more than one error.
Refer to caption: x5.png

**TABLE IV: Fine-grained errors across 287 responses to 100 programs. For select Code understanding errors related to specific code structures, the total number of responses containing those structures and the percentage with an error are shown.**
| Error category | Error | Count |
| --- | --- | --- |
| Code understanding | Bounds/NULL check | 76/159 (48%) |
| String operation | 3/29 (10%) |  |
| Integer math | 8/96   (8%) |  |
| Pointer operation | 9/147   (6%) |  |
| Alloc/Free operation | 4/81   (5%) |  |
| Index operation | 1/60   (2%) |  |
| Control flow | 11 |  |
| Syntax | 5 |  |
| Data type | 1 |  |
| Total | 118 |  |
| Hal/Mem/Rep | Hallucination | 15 |
| Memorization | 11 |  |
| Repetition | 5 |  |
| Total | 31 |  |
| Logic error | Implication | 14 |
| Inconsistent | 13 |  |
| Total | 27 |  |
| Common knowledge | Total | 8 |

We collected quantitative data by analyzing a total of 287 eligible LLM responses to understand how often LLMs make such mistakes.
To mitigate subjectivity, we used independent ratings from three authors, similar to Islam et al. [].
The raters independently identified the errors in the LLM responses, meeting to refine the protocol after processing $\nicefrac{{1}}{{3}}$, $\nicefrac{{1}}{{2}}$, and all of the data.
We measured inter-rater agreement using Fleiss’ kappa ($\kappa$) [], ranging from -1 to 1 with higher values indicating more agreement.
We resolved disagreements by siding with the majority when two raters differed from the third, resulting in $\kappa=0.75$ with 83% of responses classified identically. We then discussed the remaining disagreements until reaching a final categorization for each response.

#### III-C 2 RQ3 Results

Figure: Figure 5: Example of Missing Bounds/NULL Check.

Figure: Figure 6: Example of Incorrect Understanding of Integer Math.

Figure: Figure 7: Performance of the best prompting method/seed. Prediction displays binary prediction accuracy; bars with Bug Type, Location, and Cause display correct predictions for explaining a vulnerability. GPT-3.5, GPT-4, and Gemini each produced fewer than 5 eligible explanations each and are excluded.
Refer to caption: x6.png

Figure 4 reports how often each type of error occurred in LLM responses. We found that LLMs were somewhat useful for explaining vulnerabilities as 43% responses contain no errors. Improving Code Understanding will help LLMs to better explain vulnerabilities, as 41% responses contained code understanding errors. Our results also demonstrate that LLMs still cannot perfectly solve reasoning problems, as some of the incorrect code understanding is due to not being able to
reason about the values across several statements, and there are 9% Logic errors and 3% Common Knowledge errors, which can affect the reasoning. We observed that LLMs tended towards Hallucination, Memorization, and Repetition (11% of errors) during vulnerability detection.

In Table IV, we show the specific code structures that affected the correctness of the explanations as well as the fine-grained classifications for Logic errors and Hallucination/Memorization/Repetition errors. For example, out of 159 responses explaining bounds checks, 76 are incorrect. Figure 5 shows an example of a model response which flags an unchecked null-pointer dereference, presumably at line 5 (p->lineinfo[oldpc]), even though p->lineinfo is checked at line 2.
This error can lead to false positives, especially after the bug is fixed by adding a check.

Figure 6 shows an example of an integer math error. From the model response, the LLM correctly identified that there is a bounds-check at line 6 (it was in fact added by the developers to prevent overflows []). Given the upper bound of 0x7fff for nSamples+1 and nPatches+1, even the maximum values would not cause an overflow in an unsigned integer ($\texttt{0x7fff}*\texttt{0x7fff}*\texttt{8}=\texttt{0xfff80008}$) at line 9; however, the LLM was not able to understand and reason about such math.

Figure 7 presents the second part of the results of RQ3. Here, we provide a summary of the models’ capability of reporting the type, the location and the root cause of a vulnerability. The results show that although most LLMs correctly predicted up to 79% of vulnerable examples (blue bar), correctness dropped substantially when we consider the predicted location, bug type, and causes.
Considering both bug types and locations (orange bar), all models dropped in correctness, decreasing by 18-67%, with Mistral failing on all examples.
Considering root-cause explanations (purple bar), model correctness decreased by 24-50%.

Summary for RQ3:
57% of LLM responses contained errors when explaining the cause of vulnerabilities.
The most common error categories were “Code understanding” (41% of responses, especially missing Bounds/NULL Checks) and “Hallucination/Memorization/Repetition” (11% of responses).
All LLMs experienced a substantial (18-100%) drop in correctness when considering their predictions of bug location, type and causes.

### III-D RQ4: How do LLMs compare to human developers for localizing vulnerabilities?

#### III-D 1 Study setup

We are interested in how LLMs’ performance compares to human’s. To set up the experiments, we used DbgBench [], where the detailed human performance is reported. In their study [], 12 professional software engineers were given the source code, bug reports, and test cases related to 27 bugs in the find and grep utilities, such as null-pointer dereferences, buffer overflows, resource leaks, and infinite loops. They reported a diagnosis with fault locations, rated their perception of the difficulty, and provided a patch. To match the human setting, we gave each LLM prompts containing the bug report, test case, the source code of the files containing the faults, and instructions “Given the above bug report, test case which reproduces the bug, and source code, provide the line numbers of all fault locations in the code in the following format: * file.c:line number.”
These prompts (averaging 19-25k tokens) exceeded the context length of 4 models, which we excluded.
We extracted fault locations from the outputs of the remaining 7 LLMs and compared them with the correct fault locations from DbgBench, considering a prediction correct if at least one fault location matched the ground truth.

Table: TABLE V: LLMs correctly located 6 out of 27 DbgBench bugs. Human performance reports the average perceived difficulty and time to diagnosis, and the percentage of human participants which correctly localized each bug. 2 models failed to localize any bugs (not shown), including GPT-4.

#### III-D 2 RQ4 Results

Table V shows that among the 7 models, 5 models produced correct output for 6 bugs and made a total of 11 correct predictions (some models predict correctly on the same bug). GPT-3.5 performed the best and correctly located 4 bugs. Interestingly, the newer model, GPT-4, did not match the ground-truth fault locations on any examples, and neither did Mistral. These bugs took humans 15 to 50 minutes to diagnose, with difficulty ratings ranging from slightly to very difficult (2-4 on a scale of 1 to 5).
Figure 8 shows where these bugs located in the entire dataset in terms of perceived difficulty and diagnostic time given by DbgBench.

Figure: Figure 8: The distribution of time and difficulty reported by humans in DbgBench, and the portion of examples predicted correctly by at least one LLM.
Refer to caption: x7.png

Figure: Figure 9: Comparison of Human and LLM performance on all bugs in DbgBench. Bugs are ordered along the X-axis by human performance/diagnostic time. Y-axis indicates the percentage of humans or LLMs that correctly localized the bug.
Refer to caption: x8.png

In Figure 9, we visualize individual bugs and rank them based on the “easiness” of diagnosis for humans. To do so, we first rank the bugs based on the correctness achieved among the developers who diagnosed the bug, and to break ties, we compared the diagnosis time. Here, we can see that LLMs failed to correctly predict the examples which were the hardest for humans (rightmost dots), as well as many of the easiest examples (missing 4 out of 7 leftmost dots which were localized by 100% of humans). Overall, LLMs performed substantially worse than humans. Most of the correct predictions were among those already correctly diagnosed by 70-100% of human study participants. These findings suggest that current LLMs still have significant limitations in localizing software bugs in a real-world setting.

Summary for RQ4:
LLMs correctly located 6/27 bugs from DbgBench, while at least one human correctly located 27/27 bugs. GPT-3.5 performed the best and correctly located 4 bugs. GPT-4 and Mistral failed to locate any bugs correctly.

## IV Threats to Validity

Better prompts than those we evaluated may exist.
To draw out the best model performance, we systematically explored several prominent prompting techniques and introduced three new approaches motivated by our domain knowledge (Section III-A).
Model performance can be reduced by irrelevant context [], or the order [] and quality [] of in-context examples. To alleviate these potential issues, we reported the best-case performance among various prompt orderings and varying amounts and sources of context, and tried different number of shots (Section V).

We queried 100 functions due to the cost of LLMs (see Section II-A), which may seem limited. However, we note that our dataset originates from 36 open-source projects with diverse applications, including databases, servers, and interpreters.
The vulnerabilities represent important issues, all featuring in the top-25 most dangerous CWEs for the past five years [], and were manually curated to ensure quality [] and focus on real-world use cases []. We performed a pilot study which suggests that the models did not memorize the labels (Section V).

To mitigate potential biases from the experimental setup and metrics, we evaluated the LLMs in two distinct scenarios: “paired” and “unpaired” (Section III-B), and aggregated results across 3 different random seeds. We justify our choice of metrics in Section II-D.

The analysis of the LLMs errors can involve subjective judgments. To ensure confident classifications, we rigorously designed the categories to be unambiguous and confirmed agreement among multiple raters (Section III-C).

Our fault localization setting for DbgBench does not perfectly match the human debugging scenario, as the LLMs could not run tests or use debuggers. While such tools could aid LLMs in making a diagnosis, we focused on a direct use of LLMs in this study. Investigating long-context integration and interactive agent environments is an interesting direction for future work.

## V Discussions

Because SVEN is open-source, the labels may be inside the training dataset of recent LLMs. To test if LLMs memorized vulnerability labels, we refactored the code (using NatGen []) while preserving the vulnerability labels and evaluated the models.
If the models simply memorized the vulnerability labels, we would expect performance to be substantially reduced on the refactored code. We saw both improvements and reductions in performance and only small differences in Balanced Accuracy (0.02-0.09), even with extensive refactoring.

As a pilot study, we tried using 2, 4, 6, or 8 in-context examples for the models which performed best with in-context prompts and found that 6-shots performed best for most models.
Extended results for both experiments can be found in our data package [].

## VI Related Work

LLMs [, , , , , , , , ] have been applied in various software engineering tasks, such as code generation [], test generation [], and code summarization []
through the use of prompting.
Effective prompts can condition the model for a specific task like Question/Answer [], provide additional information through in-context examples [] and elicit step-by-step reasoning through chain-of-thought examples [, ].
We have surveyed these state-of-the-art prompting methods and applied them to the vulnerability detection task.

Recent studies have initiated investigation into the usage of LLMs for vulnerability detection, using well-known techniques such as zero-shot prompting  [, ], in-context learning [, , ], and fine-tuning [, , ].
Several papers have utilized chain-of-thoughts (CoT), such as “Let’s think step-by-step” [, , ], multi-step prompts [, ], and generic information such as CFG, DFG, PDG, and API calls [, , , ].
In this work, we propose three novel prompt approaches tailored for vulnerability detection, integrating information from bug-fix commits (contrastive pairs), CVE descriptions (CoT-CVE), and static analysis reports (CoT-SA). We reported the capabilities of LLMs to distinguish buggy and patched versions of code.

Several concurrent papers have taken initial steps to analyze errors in LLM-generated vulnerability detection responses (to the best of our knowledge, the papers are in arXiv but not yet published).
Yu et al. [] analyzed 82 responses and identified problems based a taxonomy of errors in ChatGPT responses to StackOverflow and general-purpose queries, corroborating our findings that the models struggled with correctness, logic and consistency.
Nong et al. [] categorized 100 incorrect predictions from GPT-3.5 into Insufficient Context, CWE Oblivion, Incomplete Control Flow/Data Flow; similarly, Sun et al. [] used an LLM guided by 300 human annotations to categorize incorrect predictions into Wrong Reasoning, Need Detailed Analysis, Need Other Code, and Other.
Ullah et al. [] automatically compared GPT-4’s summarization of LLM reasoning with human-generated reasoning based on BLEU, ROUGE, and GPT-4.

Our error classifications provide more fine-grained categories than prior work (see Table IV) and allow us to analyze the code structures which affected the correctness of explanations, and thus potentially provide better guidance for future improvement of LLMs for vulnerability detection. We provide a quantitative comparison of LLMs and human’s performance on locating bugs. To the best of our knowledge, our work is the most comprehensive: compared to prior works, we analyzed more models (11), and used multiple raters with a rigorous agreement protocol based on Fleiss’ $\kappa$ to manually annotate a larger set of 287 LLM responses.

## VII Conclusions

We comprehensively evaluated the vulnerability detection capabilities of LLMs from the standpoint of their performance as well as their errors.
We found that they performed only slightly better than random guessing (0.5-0.63 Balanced Accuracy), and that they struggled to differentiate buggy and fixed code versions, making the same prediction for 76% of pairs.
We proposed CoT from Static Analysis and Contrastive pairs, which improved the performance of some models.
LLMs frequently made Code Understanding, Common Knowledge, Hallucination, and Logic errors when explaining vulnerabilities, with 57% of responses containing errors.
On complex debugging tasks from DbgBench, LLMs performed far worse than humans, correctly localizing only 6/27 bugs.
These findings highlight LLMs’ limitations in vulnerability detection and our dataset of LLM errors provides guidance for future improvement of LLM-based vulnerability detection.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Do the authors have a breakdown of their results in terms of vulnerability type?
- Have they repeated this experiment on newer LLMs such as Sonnet 3.5, o1, and gpt4o? If they have seen reasoning issues it would be interesting to see o1 and sonnet 3.5 results in particular.
soundness: 3
strengths: - The paper is targeting an important problem. With LLMs being readily available for various AI for Code scenarios, whether they are a good tool to use for vulnerability detection is an important problem.
- The paper targets a reasonable number of LLMs for evaluation.
- The paper first establishes that LLMs are not good at vulnerability detection, then it tries to understand where the LLMS fail, whether its localization or reasoning. 
- The paper also analyzes whether the LLM size and training routine, as well as giving additional information from static analyzers matters in vulnerability detection.
summary: The paper evaluates LLMs capability in detecting Vulnerabilities in Code. It evaluates LLMs capabilities from various aspects, for example studying whether the LLM size and training process has any effect on its capability in detecting vulnerable code patterns.
weaknesses: - The paper concludes that LLMs are not great at vulnerability detection based on a study on SVEN, targeting C/C++ issues. While C/C++ is one of the important languages when it comes to vulnerability detection, previous research shows that even in code generation tasks LLMs perform better in more common languages such as python and javascript than C/C++. I'm not sure that for us to claim that LLMs are  in capable of detecting vulnerabilities, is it enough to look into C/C++ only. I expect that LLMs would perform better in easier to interpret languages such as JavaScript or on vulnerability patterns that are easier to catch such as "Hardcoded credentials.". Overall, a more fine-grained breakdown on the language and type of vulnerability is needed to better understand where the LLMs can be helpful and where they cannot be helpful.
- While I find the comprehensive study on different LLMs ability to find vulnerabilities insightful, none of the follow up experiments where able to point to a particular aspect that would explain why LLMs may not perform well on detecting certain vulnerabilities. It would be great to have some suggestions/insights on training routines, fine-tuning datasets, etc that can help improve LLMs performance.

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. Can you expand on the evaluation section as discussed above?
2. How does this work compare with other recent studies on vulnerability detection, such as Steenhoek et al. (2024)?

[1] Benjamin Steenhoek and Md Mahbubur Rahman and Monoshi Kumar Roy and Mirza Sanjida Alam and Earl T. Barr and Wei Le: A Comprehensive Study of the Capabilities of Large Language Models for Vulnerability Detection,CoRR, 2024


Minor comments:
Can you expand on the experimental setup in the Secion 3? It isn’t clear to me the evaluation methods for evaluating understanding and localizing errors.
soundness: 2
strengths: - The paper studies an interesting problem and pinpoints a critical challenge with great potential.
- It provides a comprehensive analysis based on the recent LLM families and demonstrates that common strategies don’t enhance LLM performance.
summary: This paper provides an in-depth analysis of LLMs' limitations in vulnerability detection, identifying the complexity of the task and the need for future work to enhance LLM’s code reasoning abilities. It further demonstrates that common strategies for enhancing LLM performance—like increasing model size, expanding training data, fine-tuning, and leveraging domain knowledge —do not significantly improve their vulnerability detection capabilities.
weaknesses: - To my understanding, the paper pinpoints the challenge of vulnerability detection across LLMs but doesn’t clearly articulate how future work could leverage the findings presented here to further improve its performance. The main contributions aren’t clearly specified.
- The additional value of breaking down different vulnerability issues into three stages is somewhat unclear and lacks evaluation. And also why do LLMs perform notably worse on tasks involving NULL checks compared to other scenarios? Offering specific suggestions for overcoming these limitations would strengthen the paper.
- The paper can benefit from extending its fine-tuning experiments beyond StarCoder2, especially by fine-tuning the SVEN and PrimeVul datasets with a broader range of LLMs.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: Overall, this work demonstrates a professional understanding of LLM research for vulnerability detection, offering meaningful insights into the limitations of current models and practical implications for their use in code security. I am ready to defend my assessment and may consider increasing my score depending on responses to the following questions.

1. Could the authors provide further justification for their chosen in-context learning design, including any tuning or design choices made for specific prompts?

2. What limitations did the authors encounter when expanding the scope of vulnerabilities? How could future work address this limitation?

3. Did the authors consider additional metrics for evaluating semantic accuracy or reasoning consistency in model responses?
soundness: 3
strengths: Major Strengths

1. Solid study on vulnerability detection limitations in LLMs: This paper presents a rigorous analysis of LLM performance in vulnerability detection, specifically on tasks requiring nuanced code reasoning. It identifies critical failure modes, such as incorrect handling of bounds checks and null-pointer dereferences. The authors highlight these challenges through detailed case studies, including examples like the pointer dereference and buffer overflow vulnerabilities, where models fail to correctly interpret safety checks or arithmetic constraints (e.g., the issues shown in Figures 1A and 1B).

2. Comprehensive evaluation on LLM-based vulnerability detection: The paper evaluates 14 models on the SVEN dataset[1], comparing baseline and advanced prompts like Chain-of-Thought from CVE (CoT-CVE) and static analysis-derived prompts (CoT-StaticAnalysis). The extensive prompt strategy testing, including contrastive pairs and static analysis paths, shows the impacts of these prompts on model performance. Despite attempts with diverse prompts, none of the strategies led to significant improvements beyond the random-guessing baseline, reinforcing the paper’s argument that current LLM capabilities are inadequate for complex vulnerability reasoning

3. Insightful analysis of failure patterns: The paper categorizes and quantifies specific recurring errors, including failures in bounds checks (50% of bounds-related errors) and misinterpretations of pointer or arithmetic operations. This error breakdown, as detailed in Table 3, clarifies the specific reasoning steps where models consistently fail, such as misunderstanding variable constraints and execution order. These insights offer concrete areas for improvement, particularly in recognizing key programming structures relevant to security​.

Minor Strengths

1. Excellent paper presentation: The paper is organized with logical sections and visual aids, including error distribution charts (Figure 3) and model performance comparisons (Figure 2), that help to convey complex results clearly. The figures, such as those highlighting prompt performance and specific error categories, provide readers with a quick understanding of each model’s strengths and weaknesses on vulnerability tasks​.

2. Methodical approach to prompt strategy comparisons: By systematically testing and comparing various prompt designs—including zero-shot, n-shot, CoT-CVE, and CoT-StaticAnalysis—the paper provides a nuanced view of how different prompt styles affect vulnerability detection performance. This comparison offers valuable insights into prompt engineering, especially as it reveals that even advanced prompts based on structured data (e.g., static analysis proofs) fail to achieve consistent improvements​.

3. Detailed manual inspection of model errors: The paper’s manual review of 300 LLM-generated responses allowed for a deeper understanding of specific errors that automated metrics might miss. This qualitative analysis shows that LLMs frequently misinterpret pointer safety and logical implications across multiple steps of reasoning. The authors provide concrete examples of where models misjudge safe versus vulnerable code (such as failing to recognize bounds checks on pointer dereferences), giving a more rounded view of LLM limitations in this domain
 
[1] He J, Vechev M. Large language models for code: Security hardening and adversarial testing[C]//Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security. 2023: 1865-1879.
summary: This paper investigates the limitations of large language models (LLMs) in performing vulnerability detection tasks, highlighting the distinct reasoning and code understanding required for effective identification of vulnerabilities. By evaluating 14 state-of-the-art LLMs across various prompt strategies on the SVEN dataset, the authors demonstrate that LLMs generally underperform in this area, particularly with tasks requiring deep code semantics, such as handling pointer operations and bounds checks. The paper offers a comprehensive breakdown of the common failure patterns observed, emphasizing the challenges LLMs face in reliably detecting code vulnerabilities and identifying specific reasoning deficiencies in current models.
weaknesses: Major Weaknesses

1. Oversimplied in-context learning: Although the authors assess several in-context learning techniques, they do not fully explore the design rationales for each prompt or the limitations that different choices present. For example, in the CoT-StaticAnalysis prompt, the inclusion of static analysis paths from the D2A dataset was intended to help models follow logical steps, but it lacks a discussion on why certain paths were prioritized over others. Adding such detail would clarify prompt-specific limitations and help in identifying where these prompts fell short for vulnerability detection.

2. Limited exploration of RAG-based systems for code-specific semantics: While the paper mentions that Retrieval-Augmented Generation (RAG) systems could benefit vulnerability detection, it does not explore how RAG might be applied to enhance the model’s understanding of complex code reasoning tasks. Incorporating RAG systems could allow models to retrieve relevant code snippets or documentation, potentially aiding in cases where semantic context (e.g., specific variable usage patterns or documentation on safe handling of pointers) is essential for accurate reasoning[1-2]. Exploring this could offer a deeper understanding of model limitations and point to meaningful directions for future work​.

Minor Weaknesses

1. Sampling may be biased: The manual inspection of 300 samples may introduce bias, as the authors do not fully detail the criteria used to select these samples or ensure diversity. Explaining how they selected these samples (e.g., random selection, specific focus on certain vulnerability types) would strengthen the credibility of their findings and ensure that the review represents the dataset as a whole.

2. Vulnerability types are limited: The study primarily examines vulnerabilities related to bounds checks, null pointers, and pointer handling, which limits the generalizability of the findings. Including additional vulnerability types, such as integer overflow (as shown in the simple example from CWE in Figure 10), would provide a broader view of LLM limitations in software security tasks.

3. Absence of advanced evaluation metrics: While the Balanced Accuracy metric is helpful in assessing basic performance, it does not capture important aspects like semantic accuracy or reasoning consistency. Introducing metrics that focus on these qualities would provide a more comprehensive evaluation of model performance on tasks requiring deep semantic understanding, such as detecting specific security flaws in code​.

[1] Du X, Zheng G, Wang K, et al. Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG[J]. arXiv preprint arXiv:2406.11147, 2024.

[2] Fayyazi R, Trueba S H, Zuzak M, et al. ProveRAG: Provenance-Driven Vulnerability Analysis with Automated Retrieval-Augmented LLMs[J]. arXiv preprint arXiv:2410.17406, 2024.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - The authors mentioned the newest PrimeVul dataset, and used it for fine-tuning. So why do the authors used the SVEN dataset for evaluation instead?

- In section 3, how are the 300 manually inspected samples selected?

- In section 3.3, the authors used the COT-annotation prompts (with static analysis) to provide additional knowledge. How is the COT-annotation prompt different from other prompts in section 2, especially the COT-StaticAnalysis prompt?
soundness: 2
strengths: +: The paper points out that vulnerability detection is a challenging task for LLMs and has not been successfully addressed by existing models. This introduces new research opportunities in both software engineering and LLM.

+: The authors defined a three-step reasoning framework for better analysis in LLMs for vulnerability detection.
summary: In this paper, the authors performed an empirical study of using LLMs for software vulnerability detection. The authors formulated vulnerability detection in 3 steps: locating potentially vulnerable statements, understanding vulnerability-related features, and predict the final vulnerability label with multi-step reasoning. The authors conducted experiments on the SVEN vulnerability dataset with LLMs under basic and COT prompts. From the results, the authors find that existing LLMs cannot effectively detect vulnerabilities, achieving results similar to random guesses. Moreover, scaling up model parameters, fine-tuning with domain-specific data, and COT prompting do not significantly improve model performances.
weaknesses: -: In section 2, the authors adopted the BigVul and D2A datasets for building prompts. However, due to a previous study [1], these datasets have high noise rates. The authors should ensure the correctness of these generated prompts.

-: The experiments should contain some of the newest LLMs, e.g., GPT-4o-mini and Llama 3.1.

-: The authors pointed out the difficulties of vulnerability detection using LLMs. Perhaps it is also better to discuss possible solutions to these difficulties.


References:

[1]: Croft, R., Babar, M. A., & Kholoosi, M. M. (2023, May). Data quality for software vulnerability datasets. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) (pp. 121-133). IEEE.

Meta-review: Summary
=======
The paper evaluates the capability of LLMs in detecting code vulnerabilities, particularly focusing on C/C++ code. The key finding is that current LLMs perform poorly at vulnerability detection, achieving results similar to random guessing. Common improvement strategies like increasing model size, expanding training data, fine-tuning, and leveraging domain knowledge do not significantly enhance performance. 

Strengths
=======
* One of the most comprehensive evaluations of LLMs thus far, using 14 state-of-the-art LLMs using various prompt strategies
* Good analysis of failure patterns with detailed categorization of error types
* Systematic testing of different prompt designs (zero-shot, n-shot, CoT-CVE, CoT-StaticAnalysis)

Weaknesses
==========
* Limited scope by focusing only on C/C++ vulnerabilities, potentially missing insights from other programming languages
* Insufficient exploration of RAG-based systems that could enhance code-specific semantic understanding
* Potential sampling bias in the manual inspection of 300 samples
* Lack of concrete suggestions for improving LLM performance in vulnerability detection
* Use of potentially noisy datasets (BigVul and D2A) for prompt construction
* Missing evaluation of some newest LLM models

Reasons for decision
================
The paper is claimed to be the most comprehensive evaluation to date (previous and concurrent evaluations point to the same conclusion: Current form of LLMs are limited for vulnerability detection). Its limited scope and other weaknesses are certainly the areas of improvement, especially the missing of solutions (which is, to be fair, not required for this type of evaluation paper).

Decision: Reject