## 3 Materials and Methods
### Model Architectures
#### 3.3.1 Deep Q-Network  
DQN is an off-policy reinforcement learning algorithm that combines Q-learning with deep neural networks. According to [PERSON] et al. [37], DQN agents can often be simulated through the probability distribution function derived from the state-value function in the action space or the scoring rules of the action-value function in the action space. This study used agents that rely on action-value functions to define the vulnerability detection logic. Here, the agent can be mathematically modeled by \(Q(s_{t},a_{t})\), where \(a_{t}\) represents the action at time \(t\) and \(s_{t}\) denotes the state at time \(t\). The \(Q\) agent acts like a prophet to evaluate each action in the action space based on the current state, thereby making optimal decisions at each moment to maximize the reward \(r_{t}\) during the progress. In the Q-learning algorithm, the agent is formed using a Q-table to store all actions across various states. However, in a real-world problem such as vulnerability detection in smart contracts, the vast number of states poses a challenge for Q-tables to manage effectively. To address this, the improved DQN utilizes deep neural networks to approximate Q-tables for constructing agents by taking a state as input and outputting Q-values for all possible actions (Figure 3) [38]. This allows the algorithm to handle high-dimensional state spaces, such as images or graphs. The training process for agent \(Q(s_{t},a_{t})\) in Q-learning relies on the algorithm of temporal difference (TD) in Equation (1), where \(\alpha\) is the learning rate, \(\gamma\) represents the discount factor, and \(a(s_{t+1})\ =\ arg\underset{a}{\max}Q(s_{t+1},a_{t+1})\) indicates the action taken by the Q-learning agent. \[Q(s_{t},a_{t})\gets Q(s_{t},a_{t})+\alpha\Big{[}r_{t+1}+\gamma\underset{ a}{\max}Q(s_{t+1},a_{t+1})-Q(s_{t},a_{t})\Big{]} \tag{1}\]  
Figure 3: Model architecture of Q-learning and DQN. In the DQN framework, the TD equation is employed to update the parameters of the neural network. Instead of learning from consecutive samples, DQN uses a replay buffer to store the agent's experiences. Specifically, the replay buffer records state transitions, selected actions, and the corresponding immediate rewards from each agent-environment interaction, forming a tuple \(\langle s_{t},a_{t},r_{t},s_{t+1}\rangle\). During training, mini-batches of experiences are randomly sampled from the buffer to update the Q-network. This technique helps to break the correlation between consecutive samples and leads to more stable training. This approach leads to the definition of the DQN objective function as in Equation (2), where \(\theta\) is the vector of parameters of agent \(Q\). \[J(\theta)\ =\ E\Big{[}(r_{t+1}+\gamma\ max_{a}Q(s_{t+1},a_{t+1};\theta)-Q(s_{t},a _{t};\theta))^{2}\ \Big{]} \tag{2}\]  
In this paper, the procedure for employing DQN agents in vulnerability detection is outlined as follows: begin at time \(t\ =\ 0\), the pair set of CFG's node and edge is initialized and fills in the data inputs. An action \(a_{0}\) is randomly selected to establish the initial state \(s_{0}\). Subsequently, for \(t\) in \([1,T]\), the agent chooses the action \(a_{t}\) that yields the highest score based on the function \(Q(s_{t},a_{t};\theta)\) and executes this action within the environment. Following this, the agent receives the immediate reward \(r_{t}\) from the environment and observes the subsequent state \(t+1\), updating the agent's parameter vector \(\theta\) accordingly, leading to the formation of the next state \(s_{t+1}\). This procedure is repeated until the environment reaches the terminal state \(T\). The primary objective of the agent is to maximise the cumulative return of \(r_{t}\). The specific algorithm for using a neural network to train the agent can be found in Algorithm 1.