# Automated Vulnerability Exploitation Using Deep Reinforcement Learning
## 2 Background
### Reinforcement Learning Fundamentals
#### 2.1.2 RL Using Deep Neural Networks  
While the Q-Table is a very efficient method to store the estimated Q-value for each state-action pair, it faces performance problems when dealing with large or continuous state spaces. Storing large states' data overwhelms the memory, and exploring every single combination becomes impractical. Such problems are addressed by Deep Q-Network (DQN), where Q-Table is replaced by NN. The NN takes the state as input and outputs the Q-values for all possible actions. The DQN is capable of generalizing from the learned patterns to explore unseen states and it has many useful applications like speech recognition, natural language processing, and malware detection [6; 7]. However, DQN designers should be aware that training can be computationally expensive and susceptible to overfitting. The standard DQN employs a single NN, which replaces the Q-Table. However, single-NN DQN can experience stability issues during training. To address this, DQN actually leverages two NNs: a policy network and a target network, as shown in Figure 1. The policy network is responsible for making decisions based on the current state of the environment. It takes the state as input and outputs a Q-value for each possible action. The agent then chooses the action with the highest Q-value. The target network is a more stable version of the policy network. It is periodically updated with the weights of the policy network, but not at every training step. This helps to smooth out the learning process. In the DQN model, the agent interacts with the environment through actions and receiving rewards. Each interaction creates an experience: \(e_{t}=(s_{t},a_{t},r_{t+1},s_{t+1})\)  
where  
* \(e_{t}\): experience/interaction at time step \(t\);
* \(s_{t}\): state at time step \(t\);
* \(a_{t}\): action at time step \(t\);
* \(r_{t+1}\): reward at time step \(t+1\);* \(s_{t+1}\): next-state (i.e. state at time step \(t+1\)). Also, a DQN model uses a replay memory, which stores the agent's experiences. The following steps are executed during training:
* A small random sample of experiences is sampled from the replay memory. * For each experience,
* The target network estimates the optimal Q-value for the next-state from the sampled experiences. This is performed by applying \(s_{t+1}\) as input to the target network and then selecting the maximum \(q(s^{\prime},a^{\prime})\). The result is the target Q-value (i.e., optimal Q-value). * The policy network computes the estimated (i.e., predicted) Q-value (i.e., \(q(s,a)\)). This is achieved by applying \(s_{t}\) as input to the network and then selecting the Q-value corresponding to \(a_{t}\). * The difference (loss) between the predicted Q-value from the policy network and the target Q-value is calculated. * The weights of the policy network are adjusted to minimize the loss, effectively making the policy network's Q-value predictions closer to the more stable target Q-values. * Occasionally, the weights of the target network are updated to match those of the policy network. This ensures the target network does not become too outdated. It is important to mention that in this work, we opted for a single NN for policy learning. Employing two NNs offered minimal performance improvements compared with the increased complexity, and we did not experience any instability issues during training. Additionally, to maintain simplicity in our DQN implementation and as Figure 1 shows, we opted for the well-established loss function, denoted as _Loss_, presented in Equation (2):  
\[Loss=E\Bigg{[}\bigg{(}\Big{[}R+\gamma\max_{A^{\prime}}Q\big{(}S^{\prime},A^{ \prime}\big{)}\Big{]}-Q(S,A)\bigg{)}^{2}\Bigg{]} \tag{2}\]  
where  
Figure 1: Deep Q-Networks (DQN). \(Q(S^{\prime},A^{\prime})\) represents the estimated future reward for taking action \(A^{\prime}\) in the next state \(S^{\prime}\), as predicted by the DQN network. * \(E\) denotes the arithmetic mean.