## 4 Experiments
### Evaluation Metrics  
To comprehensively assess the performance of the API-generated outputs, a set of evaluation metrics is employed, including Precision, Recall, Accuracy, F1-Score, Cohen's Kappa, and Matthews Correlation Coefficient (MCC). The definition of Precision and Recall is given as follows:  
\[\left\{ \begin{array}{l}\text{Precision}=\frac{\textit{TP}}{\textit{TP}+ \textit{FP}}\\ \text{Recall}=\frac{\textit{TP}}{\textit{TP}+\textit{FN}}\\ \text{TPR}=\frac{\textit{TP}}{\textit{TP}+\textit{FN}}\\ \text{FPR}=\frac{\textit{FP}}{\textit{FP}+\textit{TN}}\\ \end{array} \right. \tag{1}\]  
Here in Equation (1), _TP_ denote correctly identified vulnerable instances, while _TN_ refer to correctly classified non-vulnerable instances. _FP_ occur when a non-vulnerable case is misclassified as vulnerable, and _FN_ arise when a vulnerable instance is incorrectly predicted as non-vulnerable. Precision quantifies the reliability of positive predictions by evaluating the proportion of correctly identified vulnerabilities among all predicted positives. Recall and True Positive Rate (TPR) measures the model's ability to correctly detect actual vulnerabilities. Based on these scalar metrics, Receiver Operating Characteristic (ROC) and Precision-Recall (PR) curves are incorporated to visualize and analyze the model's performance. The ROC curve plots TPR against False Positive Rate (FPR) to illustrate the balance between sensitivity and specificity. A model with strong discriminative power produces a ROC curve that quickly approaches the top-left corner. The Area Under the ROC Curve (AUC-ROC) quantifies this performance; values closer to 1 indicate excellent classification ability, while values near 0.5 imply random guessing. The PR curve, on the other hand, focuses specifically on the trade-off between Precision and Recall. This makes it particularly insightful in scenarios involving class imbalance, such as vulnerability detection, where vulnerable cases are rare. Unlike the ROC curve, the PR curve excludes true negatives and therefore provides a more focused evaluation of a model's effectiveness in detecting positive cases. The area under the PR curve, referred to as mean Average Precision (mAP), serves as a comprehensive summary metric, where higher mAP values indicate stronger capability in identifying true vulnerabilities with minimal false alarms. F1-Score, defined as the harmonic mean of Precision and Recall, offers a balanced metric, especially useful when dealing with imbalanced class distributions, where managing the trade-off between false positives and false negatives is critical. Accuracy provides a general indicator of performance by measuring the proportion of correctly classified instances over the total number of predictions. These metrics are formally defined as follows:  
\[\left\{ \begin{array}{l}\text{F1\_Score}=\frac{2\times\text{Precision\_Recall}}{ \text{Precision\_Recall}}\\ \text{Accuracy}=\frac{TP+TN}{TP+FP+TN+FN}\end{array} \right. \tag{2}\]  
[PERSON]'s Kappa is a statistical measure that evaluates the level of agreement between predicted and true labels, while adjusting for the agreement that could occur by chance. This makes it particularly robust in scenarios involving class imbalance, where conventional Accuracy may overestimate performance. In contrast to raw Accuracy, Kappa offers a more calibrated assessment of the model's true predictive power. Its formulation is defined as follows:  
\[\left\{ \begin{array}{l}p_{o}=\frac{TP+TN}{TP+FP+FN}\\ p_{e}=\frac{(TP+FP)(TP+FN)+(FN+TN)(FP+TN)}{(TP+TN+FP+FN)^{2}}\\ \text{Kappa}=\frac{p_{e}-p_{e}}{1-p_{e}}\end{array} \right. \tag{3}\]  
In Equation (3), \(p_{o}\) represents the observed agreement, which defines the proportion of samples where the prediction matches the ground truth, while \(p_{e}\) indicates the expected agreement, which illustrates the probability of random agreement based on the marginal probabilities of each class.