## 3 Materials and Methods
### Model Architectures
#### 3.3.1 Deep Q-Network
Despite these challenges, PPO still outperformed DQN across the accuracy, recall, and F1 metrics, suggesting that on-policy updates better accommodate the combined difficulties of sparse rewards and imbalanced classes. To place our RL approaches in context, we trained three off-the-shelf classifiers: support vector machine (SVM), random forest, and a simple multi-layer perceptron (MLP), using two scalar CFG features (number of nodes and edges). We extracted these features automatically by parsing each.dot file in our dataset and counting nodes and edges. Hyperparameters (e.g., SVM's C and kernel, number/depth of trees, MLP's layer sizes and learning rates) were chosen via five-fold cross-validation on the training split. As shown \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **DQN** & **PPO** & **SVM** &
\begin{tabular}{c} **Random** \\ **Forest** \\ \end{tabular} & **MLP** \\ \hline Accuracy & 0.1869 & **0.2947** & 0.2532 & 0.2487 & 0.2214 \\ Recall & 0.1768 & **0.1930** & 0.1750 & 0.1634 & 0.1535 \\ F1 score & 0.1452 & **0.1463** & 0.1376 & 0.1308 & 0.1250 \\ \hline \hline \end{tabular} Abbreviations: DQN, Deep Q-Network; PPO, Proximal Policy Optimization; SVM, Support Vector Machine; MLP, Multi-Layer Perceptron. The best results are bolded.

\end{table} 
Table 4: Comparison of DQN, PPO and baseline ML methods performance. in the following table, while these models reach moderate performance, PPO still leads overall, confirming the benefit of sequential policy learning on graph structures. ### Evaluation of RL Model Effectiveness

Based on the validation performance, the PPO model was selected as the optimal model and subsequently deployed on the test set, with the classification report summarized in Table 5. The accuracy stands at 30%, which is consistent with the accuracy observed on the validation set, indicating the stability of the PPO model. However, the predictive performance varies significantly across different classes. The recall for unchecked calls is notably high at 0.81, with an F1 score of 0.46, the highest among all classes. We attribute PPO's particularly strong recall on unchecked calls to the ample representation of that category in the training set. Safe class follows, achieving an F1 score of 0.19. The remaining four classes exhibit suboptimal prediction performance. This outcome is consistent with the dataset's characteristics identified during data preparation, where a pronounced class imbalance was observed, with the ratio of unchecked calls to access control classes being approximately three to one. Given that nearly half of the contracts fall within the unchecked calls and safe classes, these categories have a substantial amount of data available for training the reinforcement model, leading to better predictive performance for these classes. To illustrate PPO's typical behaviour, we manually inspected two test-set contracts. In one (0x1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0), the CFG is just a single external-call block, so PPO easily learns the clear control-flow pattern and correctly flags it as an unchecked call vulnerability. In another (0x3b7a1c2e4d5f6a8b9c0d1e2f3a4b5c6d7e8f9012), the underflow happens inside a loop whose repeated arithmetic operations span multiple basic blocks--our crude node-and-edge summary collapses that structure, PPO never \"sees\" the data flow dependency, and it misses the vulnerability. A close inspection of Table 5 shows that PPO's weakest classes, access control (F1 = 0.01), arithmetic (F1 = 0.04) and reentrancy (F1 = 0.05), all hinge on semantic patterns lost in our simple node-and-edge-count summary. Permission checks for access control typically span multiple disjoint basic blocks (e.g., owner-only branches), arithmetic underflows often occur inside loops or data-dependent branches, and reentrancy involves interleaved function entry/exit blocks. Collapsing these structures into flat counts removes the exact control and data flow cues PPO needs.