## 3 Materials and Methods
### Model Architectures
#### 3.3.1 Deep Q-Network
At the micro-level, the F1 score can be calculated as: \[\mathit{Micro\ F1}\ =\ \frac{2\times\mathit{Micro\ Precision}\times \mathit{Micro\ Recall}}{\mathit{Micro\ Precision}\times \mathit{Micro\ Recall}},\] (12) where \[\mathit{Micro\ Precision}\ =\ \frac{\sum_{i\ =\ 1}^{N}\sum_{j\ =\ 1}^{Q}(y_{ij}\times\hat{y}_{ij})}{\sum_{i\ =\ 1}^{N}\sum_{j\ =\ 1}^{Q}\hat{y}_{ij}}\] (13)
* Learning curve: The learning curve plots the model's performance (in our case the reward) against the number of training iterations or episodes. It helps to visualize the learning process, showing how quickly the model learns and how well it generalizes to new data. * Receiver operating characteristic (ROC) curve: The ROC curve is a visual tool used to assess a classification model's effectiveness by illustrating the relationship between the true positive rate (TPR) and the false positive rate (FPR). TPR reflects the percentage of actual positives that the model correctly identifies, while FPR represents the percentage of negatives mistakenly classified as positives. An ideal ROC curve would rise along the \(y\)-axis toward the upper left corner, indicating high TPR and low FPR. The area under the ROC curve (AUC) provides an overall performance metric for the model. AUC values range from 0 to 1, with values near 1 indicating superior model performance and those around 0.5 suggesting performance comparable to random guessing. In a multi-label classification task, One-Vs.-All methodology is adopted to plot the ROC curve. * Precision-Recall (PR) curve: The PR curve is particularly insightful when dealing with imbalanced datasets, where the positive class is rare. It focuses on the precision (positive predictive value) and recall (sensitivity) of the model, providing a clear view of the trade-off between these two metrics. By adjusting this threshold, different combinations of precision and recall can be achieved, which are then visualized on the PR curve. Lowering the threshold typically increases the likelihood of predicting positive instances, which may result in more false positives, thus reducing precision but raising recall. Conversely, raising the threshold makes the model more selective, leading to fewer false positives and higher precision, but potentially at the cost of reduced recall. Ideally, a model would achieve both high precision and recall, with the PR curve nearing the top-right corner of the graph. The area under the PR curve (AUC-PR) serves as a single summary metric, with higher values indicating better performance. ## 4 Experiment and Analysis

In this study, our experiments were conducted on GPU nodes from a high-performance computing facility. Each node was equipped with two NVIDIA Tesla V100 GPUs (16 GB HBM2 memory each), 64 GB of DDR4 RAM, and an eight-core Intel Xeon E5-2698 v4 CPU. Experiments in this section began by evaluating the performance of reinforcement learning models using three metrics during the validation phase, accompanied by an analysis of their learning curves. Following the evaluation, the optimal one was selected and deployed to the test set for effectiveness examination. The section concludes by discussing the experimental results. ### Evaluation of RL Model Performance

Table 4 presents the validation metrics for DQN and PPO performances. Both models underperformed in this multi-label classification task, achieving only 18.69% and 29.47% accuracy, respectively. While limited data per class is a probable factor contributing to these suboptimal outcomes--our training set comprises just 13,400 contracts across six labels--additional aspects could contribute to these suboptimal outcomes. One of them could be our state encoding since compressing each CFG to only its node- and edge-count may be too simplistic to capture the fine-grained semantic patterns necessary for distinguishing vulnerability types. What is more, our binary reward scheme (correct vs. incorrect) yields very sparse feedback, which slows convergence and can lead to high variance in value estimates.