## 3 Methodology
### Framework Overview  
The end-to-end workflow is illustrated in Figure 1 for vulnerability detection utilizing different LLMs, and it outlines the key stages from data acquisition to comparative evaluation. The workflow begins by referencing the Top 25 most dangerous software weaknesses in the Common Weakness Enumeration (CWE) project [46]. The corresponding test cases are retrieved from the Software Assurance Reference Dataset (SARD) [47].  
As shown in the Data Preprocessing Stage of Figure 1, a comprehensive pipeline is implemented to ensure data quality and consistency in the experiment. Firstly, invalid or duplicate code samples are removed. Secondly, code comments are stripped to prevent misleading contextual cues that may bias the model's decision-making. Furthermore, filenames containing sensitive or potentially identifying content are anonymized to ensure fairness and eliminate unintended cues for model inference.  
To rigorously examine the impact of reasoning on vulnerability detection performance, the Prompt Strategy module designs distinct prompting strategies for both reasoning LLMs and non-reasoning LLMs. These prompt templates are designed to align with each model's respective capabilities, facilitating a fair and effective comparison. In this study, the selected models are categorized into two groups based on their reasoning capabilities: non-reasoning models and reasoning models. Specifically, non-reasoning models such as GPT-4o-Mini, Claude 3.5, and DeepSeek-V3 typically exhibit limited multi-step reasoning capabilities and tend to rely on pattern matching or memorization rather than deep comprehension. In  
Figure 1: The proposed workflow for vulnerability detection using LLMs.  
contrast, reasoning models like O3-Mini and DeepSeek-R1 are trained with reinforcement learning and optimization techniques. They are designed to encourage multi-hop reasoning and structured thinking. These models are capable of generating intermediate steps in their responses, leading to more reliable results on complex tasks. The specific prompting strategies are provided in Section 3.4.  
In the Model Inference phase, the preprocessed code samples are fed into each LLM. This study evaluates five representative models: Claude-3.5-Haiku, GPT-4o-Mini, DeepSeek-V3, O3-Mini, and DeepSeek-R1. Model responses are obtained via official APIs, preserving both raw output and any intermediate reasoning traces when available.  
Next, the Post-processing stage, as reflected in Figure 1, involves keyword-based extraction of critical metric information, such as \"Probability of Vulnerability\", enabling consistent and quantitative analysis. To enhance robustness, this stage also incorporates multiple keyword patterns in the post-processing scripts to accurately extract final results across diverse model outputs. The data preprocessing and post-processing codes will be included at commit _main_ in the publicly released GitHub repository to facilitate understanding and reproducibility, as well as some CWE code samples.  
Finally, a Comparative Evaluation is conducted across all models based on key detection metrics. This stage not only benchmarks performance differences but also analyzes the impact of reasoning capabilities through detailed case studies. Furthermore, it discusses the potential limitations of current LLMs in software vulnerability detection.