# Automated Vulnerability Exploitation Using Deep Reinforcement Learning
## 5 Results and Discussion
### CouchDB Using Q-Table and DRL  
Figure 5 and Table 5 summarize the results for exploiting CouchDB using Q-Table. To assess the performance of the RL agent, we plotted the moving average of Q-values, which provides a smoothed representation of the agent's learning progress over time. A moving average plot for Q-values demonstrates the trend of the agent's Q-value estimates, highlighting how they change as the agent interacts with the environment and learns from its experiences. \begin{table}
\begin{tabular}{c c} \hline \hline
**Use Case** & **Execution Time (Model Training,** \\  & **Vulnerability Exploitation)** \\ \hline CouchDB using Q-Table & 2.5 h, 8.1 s \\ \hline CouchDB using DRL & 2.5 h, 8 s \\ \hline GV using Q-Table & 4.3 h, 8 s \\ \hline GV using DRL & 4.5 h, 8 s \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Execution time of model training and vulnerability exploitation. Figure 5 shows the results after running the agent for 500 episodes each time. We chose this number of episodes because running for more episodes would not improve the results any further, but it would take much longer and use more computer power. For each run, the success rate (\(SR\)) was calculated. This is simply the number of reverse shell sessions successfully established by the agent (\(x\)) out of the total number of runs (\(N\)), as shown in Equation (3). \[SR=x/N \tag{3}\]  
Finding the perfect settings for the agent's learning (\(\alpha\) and \(\gamma\)) can be tricky, but they play a big role in how well it performs. Our results (Table 5) show that starting with settings in the middle range (around 0.6) for both how much the agent learns from new things (\(\alpha\)) and how often it tries random actions (\(\gamma\)) leads to the best results. This is because this balance allows the agent to learn effectively while also exploring new possibilities. We also found that gradually reducing the random actions (exploration) over time leads to better success rates. In short, all our tests were successful after running the training for 500 episodes. This shows that 500 episodes is enough training time for the agent. On the other hand, Figure 6 and Table 6 summarize the results for exploiting CouchDB using DRL. Figure 6 demonstrates that the moving average of the rewards has already converged before 500 episodes, similarly to Figure 5. Table 6 shows interesting results for this scenario. The accuracy improved overall compared with when a Q-Table was used. This can be explained by the _continuous learning_ characteristic of the DRL agent  
Figure 5: Rewards’ moving average for CouchDB using Q-Table in the training phase for 5 scenarios where the values of \(\epsilon,\alpha,\gamma\), rate of decrease, and success rate are shown, respectively, under each figure. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Trial** & **Epsilon** & **Alpha** & **Gamma** & **Decay Rate** & **SR \%** \\ \hline
1 & \(1\times 10^{1}\) & \(6\times 10^{-1}\) & \(4\times 10^{-1}\) & \(1\times 10^{-2}\) & 80.4 \\
2 & \(1\times 10^{1}\) & \(4\times 10^{-1}\) & \(6\times 10^{-1}\) & \(1\times 10^{-2}\) & 81.4 \\
3 & \(6\times 10^{-1}\) & \(6\times 10^{-1}\) &  
and the details of these trials can be found in Table 8, where the best success rate 73.6% was achieved in Figure 8b. In Figure 8, we present the results of these trials. Figure 8a shows that the agent has achieved an optimal policy with a notable success rate, reaching 72.1% in terms of the number of successful episodes. This trial stands out as the best-performing one due to the well-balanced value of \(e\), allowing for a trade-off between exploration and exploitation. Additionally, the success rate is relatively high, indicating that the agent has effectively learned an optimal policy. Figure 8b exhibits a high success rate; however, the agent has not yet attained an optimal policy. The value of epsilon encourages the agent to prioritize exploration over exploitation, resulting in ongoing learning without fully converging to an optimal solution.