## 2 Related Works
### Vulnerability Detection with Large Language Models  
Naturally, we are not the first to attempt to use the interpretive capacity of large language models to examine program code vulnerabilities. In their 2021 literature review [30], [PERSON] presented research on software vulnerability detection based on BERT and GPT. Common among the presented efforts was the segmentation of source code, followed by the extraction of features, and the fact that the introduction of specific vulnerabilities was one of the final fine-tuning steps. [PERSON] et al. [31] investigated the efficacy of large transformer-based language models, such as GPT-2, in detecting software vulnerabilities in C/C++ source code and demonstrated that these models outperform conventional models such as BiLSTM and BiGRU on a variety of performance metrics. It is important to note that implementing these language models presents challenges, such as GPU memory constraints and the complexities of managing model parallelism and dependencies when refining and running them locally. The study by [PERSON] [32] presents VulDetect, a transformer-based vulnerability detection framework. It fine-tunes a pre-trained language model (GPT) to identify vulnerable code, achieving up to 92.65% accuracy. This outperforms current state-of-the-art techniques such as SyseVR and VulDeBERT. However, the practicality of this approach might be questioned. Traditional deep learning models like CNNs and LSTMs, which their research aims to surpass, are known for their high computational needs. This makes them less ideal for real-time applications. [PERSON] et al. [33] observed that many studies, like the ones described above, do not leverage the full capabilities of GPT-type models. These models can interpret and evaluate domain-specific information. Instead, the focus has mainly been on identifying vulnerabilities through control and data flow analysis. They presented GPTScan, which searched the source code of smart contracts for logical inconsistencies with 90% accuracy. However, while GPTScan achieves high precision for token contracts, it only achieves \"acceptable\" precision (57.14%) for larger projects like Web3 Bugs. [PERSON] et al. [34] attempted to detect the most prevalent CWE vulnerabilities in open-source Java applications in a manner similar to ours by prompting GPT-3.5 and ChatGPT. However, their work was based on rather naive assumptions about the existing knowledge set of GPT; they only mentioned prompt engineering techniques in their conclusions, and their results were not particularly convincing, performing essentially on par with their benchmarking dummy classifier. [PERSON] and [PERSON] [35] attempted a similar experiment with ChatGPT, although they have improved their approach by using chain-of-thought and few-shot example prompt engineering techniques, showing promising results in creating an Android bug replay automation tool supported by ChatGPT. Still, the current automated approaches to bug replication are limited by the limitations of manually-crafted patterns and pre-defined vocabulary lists, which determine their success based on the characteristics and quality of defect reports. A comparison of the mentioned works is given in Table 1. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Author** & **Models** & **Subject** & **Weaknesses** \\ \hline \multirow{3}{*}{[PERSON] [30]} & \multirow{3}{*}{BERT and GPT} & Software vulnerability detection & Relies on segmentation of source code and feature extraction \\ \cline{3-4}  & & Software Vulnerabilities in C/C++ source code & GPU memory constraints, managing model parallelism and dependencies \\ \hline \multirow{3}{*}{[PERSON] [32]} & \multirow{3}{*}{GPT-2} & \multirow{3}{*}{Vulnerabilities in C/C++ codebases} & High computational burden of traditional deep learning models like CNNs and LSTMs \\ \cline{3-4}  & & & \\ \cline{1-1} \cline{3-4}  & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: List of related works with the most relatable solutions to our own research.