## 4 Experiments
### Experimental Results
#### 4.3.1 Performance Comparison
The PR curve offers insight into the trade-off between precision and recall and the overall effectiveness beyond a single threshold point. It illustrates how model performance varies across a range of threshold values. As shown in Figure 6, DeepSeek-R1 achieves the highest mAP at 0.9756, with O3-Mini closely following at 0.9743. These results reinforce the effectiveness of reasoning models in accurately capturing vulnerability patterns. DeepSeek  
Figure 4: Accuracy Distribution across different LLMs. Figure 3: Accuracy and F1-Score Comparison across different LLMs. V3 also performs well with mAP at 0.9411, indicating a strong balance between reducing false positives and maintaining high recall. On the other hand, GPT-4o-Mini and Claude-3.5-Haiku register lower mAP scores of 0.8670 and 0.8364, respectively, suggesting diminished efficacy in distinguishing true vulnerabilities. The ROC curve evaluates the trade-off between the true positive rate (TPR) and false positive rate (FPR) and visualizes how model performance varies across different classification thresholds. As illustrated in Figure 7, DeepSeek-R1 again leads with an AUC-ROC of 0.9637, followed by O3-Mini at 0.9590, highlighting their robustness in binary classification scenarios. DeepSeek-V3 also demonstrates strong discriminatory power (AUC-ROC: 0.9141), while GPT-4o-Mini and Claude-3.5-Haiku exhibit lower scores (0.8194 and 0.7429, respectively), indicating reduced classification confidence. Figure 5: F1-Score Distribution across different LLMs. Figure 6: Precision-Recall Curve Comparison across different LLMs. As shown in Figure 8, the reasoning-aware model DeepSeek-R1 achieves the highest agreement score with Kappa = 0.8770 and MCC = 0.8779, indicating strong alignment between predicted and actual labels even under class imbalance. O3-Mini also performs competitively with Kappa = 0.8438 and MCC = 0.8472, suggesting high overall quality in its predictions. Models like GPT-do-Mini and DeepSeek-V3 show moderate agreement levels with Kappa from 0.69 to 0.72 and MCC from 0.70 to 0.73, while Claude-3.5 lags behind with Kappa = 0.4250 and MCC = 0.4534, reflecting lower consistency and less robust classification performance. The 95% confidence intervals using 1000-round bootstrap resampling for Kappa and MCC across different LLMs is depicted in Figures 9 and 10. These intervals  
0.8734) and MCC of (0.8173, 0.8759), and its slightly lower upper bounds and narrower margins suggest less consistent alignment with the ground truth compared to DeepSeek-R1. In contrast, GPT-4o-Mini and DeepSeek-V3 show comparable performance, but their confidence intervals (e.g., Kappa CIs of (0.6730, 0.7589) and (0.6442, 0.7277), MCC CIs of (0.6891, 0.7692) and (0.6551, 0.7349), respectively) overlap substantially, suggesting no statistically significant difference between them. However, both trail behind the previous two reasoning models by a noticeable margin, particularly in their lower bounds. Finally, Claude-3.5 records the weakest agreement scores, with Kappa and MCC wide intervals of (0.3655, 0.4787) and (0.3938, 0.5080), reflecting greater variability and a weaker correlation structure in its predictions. These confidence intervals not only reinforce the ranking among the five models but also justify the statistical reliability of DeepSeek-R1's improvements in terms of agreement strength and prediction quality. Given the relatively small performance gap between DeepSeek-R1 and O3-Mini, it is crucial to determine whether such differences are statistically significant rather than attributable to random chance. To this end, McNemar's Test [50], a non-parametric method for paired nominal data, is applied to evaluate whether two classifiers have significantly  
Figure 10: MCC Distribution across different LLMs. Figure 9: Kappa Distribution across different LLMs. different error rates on the same test instances. The more commonly used variant is proposed by [PERSON] [51], which defines the _chi_squared_statistic_ and _p_value_ as follows:  
\[\left\{