## 3 Materials and Methods
### Model Architectures
#### 3.3.1 Deep Q-Network
Safe (F1 = 0.19) and unchecked calls (F1 = 0.46) produce more uniform graphs (single-block safety idioms or lone external calls) that PPO can readily learn. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Precision** & **Recall** & **F1 Score** & **Support** \\ \hline access control & 0.03 & 0.01 & 0.01 & 193 \\ arithmetic & 0.11 & 0.02 & 0.04 & 249 \\ other & 0.15 & 0.03 & 0.05 & 410 \\ reentrancy & 0.25 & 0.03 & 0.05 & 441 \\ safe & 0.21 & 0.17 & 0.19 & 565 \\ unchecked calls & 0.28 & 0.81 & 0.46 & 693 \\ accuracy & & & 0.30 & 2534 \\ micro avg & 0.19 & 0.19 & 0.17 & 2534 \\ macro avg & 0.20 & 0.18 & 0.14 & 2534 \\ weighted avg & 0.21 & 0.28 & 0.20 & 2534 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Classification report of PPO model on the test set. To visualize the model effectiveness among various vulnerability categories, both the ROC curve and the precision-recall (PR) curve were plotted for analysis (Figure 4). The ROC curve reveals similar suboptimal model performance across six classes, with the area under the curve (AUC) values hovering around 0.55. However, in the case of imbalanced datasets where positive class is infrequent, the ROC curve can be misleading, as it might indicate high performance even if the model performs poorly on the positive class. Thus, the PR curve offers a more insightful evaluation for our task, as it focuses on the positive class (i.e., the vulnerability class) and aims to mitigate the significant costs associated with false positives (i.e., falsely detecting vulnerabilities). The results align with the classification report, as the area under the PR curve for unchecked calls stands at 0.78, indicating that the model demonstrates moderate effectiveness when predicting the majority class. ### Discussion

Despite the creative effort in applying reinforcement learning algorithms to vulnerability detection, there are still some limitations faced by this study and directions for future research. One significant limitation lies in the use of a limited dataset for training the reinforcement learning models. This limitation hinders the models' ability to adequately learn patterns across individual classes. This research primarily relies on CFGs to extract high-level semantics from contract bytecode. However, other graph representations, such as abstract syntax tree and program dependency graph, may offer richer syntactic and semantic insights. Another important direction for future work involves the integration of this approach into practical smart contract auditing tools. By embedding the reinforcement learning model into platforms such as Slither, MythX, or even custom development environments, the vulnerability detection process could be made more dynamic and adaptive. Unlike traditional static analyzers that rely on predefined heuristics, a reinforcement learning-based tool could learn from emerging patterns and improve over time. This would significantly enhance its utility in real-world development settings, offering developers real-time guidance and automated vulnerability detection throughout the contract development lifecycle. ## 5 Conclusions

This paper shows that reinforcement learning techniques could be a viable tool for detecting and classifying smart contract vulnerabilities. Utilizing a publicly sourced dataset, two distinct model architectures were trained and evaluated. Treating vulnerability detection as a multi-label classification problem, graph representations were generated to capture relevant semantic features from the contract bytecode. The results indicate that, given the configurations in terms of custom environment setup and hyperparameter tuning, the PPO model exhibits more stable and consistent learning patterns and achieves higher overall

Figure 4: PPO’s effectiveness across vulnerability categories: (**a**) ROC curve; (**b**) PR curve. rewards. When applied to the test set, the PPO model displays effectiveness, particularly for the majority class of vulnerabilities, as evidenced by metrics such as Micro Recall, Micro F1, and the PR curve.