## 4 Experiments
### Experimental Results
#### 4.3.2 Category-Wise Analysis  
To further investigate the per-type effectiveness of various LLM-based vulnerability detection methods, the model performance across different CWE categories in Table 1 is evaluated. The detailed results for both Accuracy and F1-Score are reported in Table 4, where the highest values for each CWE are marked in bold, and the second-highest scores are underlined for clarity. This fine-grained comparison provides insights into how different models generalize across diverse vulnerability patterns. As shown in Table 4, DeepSeek-R1 consistently outperforms all other models, achieving the highest average Accuracy (0.8688) and F1-Score (0.8982) across all vulnerability categories. GPT-4o-Mini emerges as the second, with an overall Accuracy of 0.8487 and F1-Score of 0.8691, often demonstrating results very close to those of DeepSeek-R1. \begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**Claude-3.5-Haiku**} & \multicolumn{2}{c}{**GPT-4o-Mini**} & \multicolumn{2}{c}{**DeepSeek-V3**} & \multicolumn{2}{c}{**O3-Mini**} & \multicolumn{2}{c}{**DeepSeek-R1**} \\ \cline{2-10}
**CWE Type** & **Accuracy** & **F1-Score** & **Accuracy** & **F1-Score** & **Accuracy** & **F1-Score** & **Accuracy** & **F1-Score** & **Accuracy** & **F1-Score** \\ \hline CWE-20 & 0.3846 & 0.5000 & 0.7692 & 0.8571 & 0.6923 & 0.7778 & 0.3846 & 0.4286 & **0.8462** & **0.9000** \\ CWE-22 & **0.7857** & **0.6667** & 0.2857 & 0.2857 & 0.3571 & 0.1818 & 0.5714 & 0.2500 & 0.3571 & 0.4000 \\ CWE-78 & 0.9225 & 0.9597 & 0.9225 & 0.9597 & 0.8062 & 0.8918 & **0.9767** & **0.9877** & 0.9297 & 0.9620 \\ CWE-79 & 0.5000 & 0.5000 & **0.5500** & **0.6667** & **0.5500** & 0.5714 & 0.3000 & 0.3636 & 0.4737 & 0.6429 \\ CWE-89 & 0.9606 & 0.9793 & 0.9213 & 0.9587 & 0.9200 & 0.9565 & **0.9921** & **0.9957** & 0.9360 & 0.9667 \\ CWE-119 & 0.5558 & 0.4903 & 0.8010 & 0.6846 & 0.9417 & 0.8788 & **0.9830** & **0.9617** & 0.9612 & 0.9175 \\ CWE-125 & 0.4000 & 0.5714 & **0.9333** & **0.9655** & 0.2667 & 0.4211 & 0.0000 & 0.0000 & 0.8667 & 0.9286 \\ CWE-190 & 0.9667 & 0.9831 & 0.9833 & 0.9916 & 0.7667 & 0.8679 & 0.9667 & 0.9831 & 0.9915 & **0.9957** \\ CWE-287 & 0.5000 & 0.6667 & **1.0000** & **1.0000** & 0.5000 & 0.6667 & 0.0000 & 0.0000 & **1.0000** & **1.0000** \\ CWE-400 & 0.9667 & 0.9831 & 0.9750 & 0.9873 & 0.8333 & 0.9091 & 0.9917 & 0.9958 & **1.0000** & **1.0000** \\ CWE-416 & 0.8768 & 0.9339 & **0.9565** & **0.9766** & 0.8478 & 0.9099 & 0.8841 & 0.9328 & 0.9489 & 0.9717 \\ CWE-476 & 0.8661 & 0.9264 & 0.9449 & 0.9694 & 0.8571 & 0.9167 & 0.9213 & 0.9550 & **0.9921** & **0.9956** \\ CWE-787 & 0.9725 & 0.9860 & 0.9908 & 0.9954 & 0.9725 & 0.9860 & 0.9725 & 0.9860 & **0.9908** & **0.9954** \\ \hline Total & 0.7429 & 0.7805 & 0.8487 & 0.8691 & 0.7163 & 0.7643 & 0.6880 & 0.6800 & **0.8688** & **0.8982** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparison of Accuracy and F1-Score for each CWE type across LLMs. To facilitate intuitive comparisons and better visualization, two heatmaps are additionally provided in Figures 11 and 12 to depict the distribution of Accuracy and F1-Score, respectively. Darker shades indicate higher metric value. It can be observed that DeepSeek-R1 consistently delivers top-tier performance across a broad spectrum of CWE types. Specifically, it achieves either the highest or second-highest F1-Score across all 13 CWE categories, demonstrating strong generalization and robust detection capabilities. In terms of Accuracy, DeepSeek-R1 ranks first or second on more than ten CWE types, further confirming its overall superiority. Both Claude-3.5-Haiku and DeepSeek-V3 exhibit less stable performance, with notable strengths and weaknesses depending on the specific CWE type. While O3-Mini ranks lower in average performance, this is largely attributed to its complete failure to correctly predict samples in two low-sample CWE types, CWE-125 and CWE-287. These misclassifications significantly skew its mean metrics downward.