## 3 Materials and Methods
### Model Architectures
#### 3.3.1 Deep Q-Network
* Steps before learning (DQN specific): This defines how many steps of the model to collect transitions for before learning starts. Delaying learning can help the model accumulate a diverse set of experiences, improving the stability and performance of the learning process. * Buffer size (DQN specific): Size of the replay buffer that stores past experiences for the model to learn. A larger buffer allows the model to learn from a broader range of experiences but may increase memory usage and slow down training. * Number of steps (PPO specific): The number of steps to run for each environment per update. More steps can stabilize the model by providing more data for each update but may also slow down the learning process. * Number of epochs (PPO specific): This parameter specifies how many times the model iterates over collected experience data to optimize the policy during each update. In the context of smart contract vulnerability detection, more epochs can help the model better generalize patterns associated with vulnerabilities by revisiting observed contract behaviors more thoroughly. However, excessive epochs can lead the model to overfit to specific contract structures or vulnerability patterns in the training data, reducing its ability to detect novel or rare vulnerabilities in real-world DeFi contracts. Therefore, a careful balance is necessary to ensure robustness without compromising generalization. * Entropy coefficient (PPO specific): Entropy coefficient for the loss calculation. A higher coefficient can promote exploration and help in avoiding local minima. * Clip range (PPO specific): Clipping parameter. It can be a function of the current progress remaining (from 1 to 0). In PPO, the clip range limits how much the policy can change during training, which helps maintain the balance between exploring new policies and retaining what has already been learned. The results of hyperparameter tuning for DQN and PPO algorithms are presented in Tables 2 and 3, where the max rewards achieved by both models are 551.0 and 584.0, respectively. (PR) curve were also implemented to evaluate the model performance across classes using One-Vs.-All methodology. * Label Accuracy: this measures the proportion of correct predictions for each label across all instances. Formally, denote \(y_{ij}\) as the true label value (either 0 or 1) of the \(j\)th label for the \(i\)th sample, and \(\hat{y}_{ij}\) as its predicted label value. Label accuracy can be defined as: \[\mathit{Label\ Accuracy}\ =\ \frac{\sum_{i\ =\ 1}^{N}\sum_{j\ =\ 1}^{Q}\delta(y_{ij}\, \hat{y}_{ij})}{N\times Q},\] (9) where \(N\) represents the total number of samples, \(Q\) is the total number of labels, \(\delta\) is the indicator function that equals 1 when \(y_{ij}\ =\ \hat{y}_{ij}\) and 0 otherwise. * Subset Accuracy: a stricter accuracy score that measures the proportion of instances where all labels are correctly predicted. The mathematical formulation is: \[\mathit{Subset\ Accuracy}\ =\ \frac{1}{N}\sum_{i\ =\ 1}^{N}\delta(y_{ij}\, \hat{y}_{ij}).\] (10)
* Micro-Recall: Recall measures the proportion of correctly identified out of all actual positives. For micro-level evaluation, labels from all samples are merged into a single comprehensive set. The recall is then calculated based on this aggregated set with the following formula: \[\mathit{Micro\ Recall}\ =\ \frac{\sum_{i\ =\ 1}^{N}\sum_{j\ =\ 1}^{Q}(y_{ij}\times\hat{y}_{ij})}{\sum_{i\ =\ 1}^{N}\sum_{j\ =\ 1}^{Q}y_{ij}}.\] (11)
* Micro-F1: The F1 score is the harmonic mean between precision and recall, bounded between 0 and 1.