## 3 Materials and Methods
### Model Architectures
#### 3.3.1 Deep Q-Network
```
Initialize policy parameter \(\mathbf{\theta_{0}}\) Initialize value function parameters \(\mathbf{\phi_{0}}\) For \(\mathbf{k}~{}=~{}\mathbf{0,1,2,\ldots}\) do: Collect set of trajectories \(\mathbf{\mathcal{D}_{k}}~{}=~{}\{\mathbf{\tau}_{i}\}\) by running policy \(\pi_{\mathbf{k}}~{}=~{}\mathbf{\pi}(\mathbf{\theta_{k}})\) in the environment Compute rewards-to-go \(\mathbf{\hat{R}_{t}}\) Compute advantage estimates \(\mathbf{\hat{A}_{t}}\) based on current value function \(V_{\mathbf{\phi_{k}}}\) Update the policy by maximizing the PPO-Clip objective: \(\mathbf{\theta_{k+1}}~{}=~{}\underset{\theta}{argmax}\frac{\mathbf{1}}{|\mathbf{\mathcal{D }_{k}}|\mathbf{T}}\sum\limits_{\tau\in\mathcal{D}_{k}}\sum\limits_{t}^{T}\mathbf{ \min}\bigg{(}\frac{\pi_{\theta}(a|s)}{\pi_{\theta_{k}}(a|s)}A^{\pi_{\theta_{k} }}(s_{t},\mathbf{a}_{t}),~{}g(\epsilon,~{}A^{\pi_{\theta_{k}}}(s_{t},\mathbf{a}_{t})) \bigg{)}\), typically via stochastic gradient ascent with Adam Fit value function by regression on mean-squared error: \(\mathbf{\phi_{k+1}}~{}=~{}\underset{\phi}{argmax}\frac{\mathbf{1}}{|\mathbf{\mathcal{D}_{k }}|\mathbf{T}}\sum\limits_{\tau\in\mathcal{D}_{k}}\sum\limits_{t}^{T}\big{(}V_{ \mathbf{\phi}}(s_{t})-\hat{R_{t}}\big{)}^{2}\), typically via some gradient decent algorithm end for
```
**Algorithm 2**PPO-Clip

### Model Configurations

#### 3.4.1 Agent Environment

To ensure the reinforcement learning agent performs well, it is necessary to set up a custom environment tailored to the classification task as formed in this paper. It involves constructing the action and observation spaces, implementing functions to retrieve the current observation and transition between states, and incorporating a function to compute the immediate reward. After setup, it is necessary to apply the training, validation and testing sets to the environment. The complete framework for environment setup is detailed in Algorithm 3. ```
class SmartContractEnv() function extract_features_from_cfg() Extract graph features from.dot file of CFGs function_init_0 Initialize current step as 0 Initialize action space as spaces. Discrete(6) to capture six classes Initialize observation space as feature shape of extract_features_from_cfg() function_get_observation() Extract the CFG features for the current contract function step() Set current step forward while episode not done, do: Set next state by_get_observation() Calculate reward by_calculate_reward() end while function_calculate_reward() Define the reward calculation rules as: when action matches true labels, get a reward 1 Initialize environments to training, validation and test sets by SmartContractEnv()
```

**Algorithm 3** Agent Environment Setup

#### 3.4.2 Hyperparameter Tuning

Hyperparameter tuning is a critical aspect of optimizing reinforcement learning algorithms like DQN and PPO because it directly influences the model's performance and convergence. In reinforcement learning, hyperparameters control various aspects of the learning process, such as how quickly the model adapts to changes (learning rate), the size and management of experience replay buffers (buffer size, etc. ), how actions are selected (gamma), and the frequency and the way updates are made (batch size, steps, epochs). In this paper, the following hyperparameters were considered for tuning:

* Learning rate: This parameter controls how much the model updates its knowledge after each step. A too-high learning rate might cause the model to overlook optimal solutions, while a too-low rate may result in slow learning and the model getting stuck in local optima. * Batch size: Size of the minibatch for training. Larger batch sizes generally lead to more stable updates but require more computational resources. * Gamma: The discount factor belonging to (0, 1) that represents the importance of future rewards. A gamma close to 1.0 makes the model value long-term rewards, while a smaller gamma makes it prioritize immediate rewards.