## 4 Experiments
### Experimental Results  
#### 4.3.1 Performance Comparison  
To ensure consistency across all evaluations, OpenAI's unified API framework is implemented to interface with each selected LLM, standardizing input parameters and output formatting. For every code sample, the model-generated answer is stored as an answer file, and if applicable, the corresponding reasoning process is saved separately as a reasoning file. Among all tested models, only DeepSeek-R1 provides a structured, step-by-step reasoning output. Hence, in the subsequent discussions, the term reasoning process specifically refers to the output generated by DeepSeek-R1. A key consideration when invoking LLMs through OpenAI's API involves understanding constraints related to context length, maximum tokens, and response limits. These limitations vary across models. For instance, GPT-4o-Mini enforces a strict cap on the maximum context length, which, if exceeded, results in an error due to surpassing token constraints. All samples are processed via OpenAI's API with a selected LLM. Each model independently generates its answer file, from which the relevant vulnerability indicators are extracted using the predefined keywords. A parsing script is employed to identify and compute probability estimates for vulnerability detection across all samples. The comparative performance metrics of the models are summarized in Table 3.  
As depicted in Figure 3, DeepSeek-R1 demonstrates the highest Accuracy (0.9507) and F1-score (0.9659), underscoring its superior performance in identifying software vulnerabilities with balanced precision and recall. Close behind, O3-Mini attains an Accuracy of 0.9338 and an F1-score of 0.9524, indicating a similar level of effectiveness. In contrast, Claude-3.5-Haiku lags behind other models in both metrics, suggesting a comparatively weaker detection capability. GPT-4o-Mini and DeepSeek-V3 perform moderately well, though they exhibit a slight drop in overall reliability. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Model** & **Accuracy * & **F1-Score** & **mAP** & **AUC-ROC** & **Kappa** & **MCC** \\ \hline Claude-3.5-Haiku & 0.7964 & 0.8690 & 0.8364 & 0.7429 & 0.4250 & 0.4534 \\ \hline GPT-4o-Mini & 0.8930 & 0.9288 & 0.8670 & 0.8194 & 0.7158 & 0.7290 \\ \hline DeepSeek-V3 & 0.8637 & 0.8998 & 0.9411 & 0.9141 & 0.6885 & 0.6973 \\ \hline O3-Mini & 0.9338 & 0.9524 & 0.9743 & 0.9590 & 0.8438 & 0.8472 \\ \hline
**DeepSeek-R1** & **0.9507** & **0.9659** & **0.9756** & **0.9637** & **0.8770** & **0.8779** \\ \hline \hline \end{tabular}  
* A default threshold of 0.5 was used for Accuracy and F1-Score.  
\end{table} 
Table 3: Comparative performance metrics across LMMs. The 95% confidence intervals derived from bootstrap resampling for Accuracy and F1-Score are depicted in Figures 4 and 5. These intervals help determine whether differences in agreement levels across models are statistically meaningful or attributable to random fluctuations in the evaluation. Among the evaluated models, DeepSeek-R1 demonstrates the highest performance, with the highest Accuracy and F1-score, accompanied by tight 95% confidence intervals (0.9395, 0.9619) for Accuracy and (0.9575, 0.9735) for F1-Score, indicating high stability and robustness. In contrast, O3-Mini also achieves strong performance results, yet its confidence intervals (0.9211, 0.9464) for Accuracy and (0.9429, 0.9616) for F1-Scores show only partial overlap with those of DeepSeek-R1. Meanwhile, models like GPT-4o-Mini and DeepSeek-V3 exhibit broader and more overlapping intervals, e.g., the GPT-4o-Mini's Accuracy CI of (0.8759, 0.9086) and F1-Score CI of (0.9167, 0.9394), making their margins of difference less decisive compared to the above two reasoning models. Claude-3.5, the weakest among the five, shows both the lowest central scores and the widest confidence bounds (0.7749, 0.8165) and (0.8531, 0.8831), indicating greater variability and less consistent performance.