# Automated Vulnerability Exploitation Using Deep Reinforcement Learning
## 4 Methodology
### RL Training and Deployment: Q-Table  
In this subsection, we discuss Q-Table's training and deployment for use cases 1 and 2. The state '\(S\)' of the system refers to the combination of an OS and a certain vulnerability. The Metasploit payloads that are used to exploit the vulnerability and change its state to a compromised OS represent the actions that can be performed '\(A\)'. Table 3 summarizes the states, actions, and RL parameters that exist in the system. In the following list, we explain how the RL parameters influence the training process:  
* Learning rate (\(\alpha\)): This controls how much weight the agent gives to new information versus past experiences. A higher value means the agent prioritizes new information, while a lower value emphasizes past experiences. * Discount factor (\(\gamma\)): This balances the value of immediate rewards (benefits right now) with future rewards (benefits later). \begin{table}
\begin{tabular}{c c} \hline \hline
**CVE** & **Description** \\ \hline CVE-2011-3556 [28] &
\begin{tabular}{c} Unspecified vulnerability in the Java Runtime \\ Environment component in Oracle Java SE JDK \\ and JRE \\ \end{tabular} \\ \hline CVE-2007-2447 [29] & The MS-RPC functionality in smbd \\ \hline CVE-2004-2687 [30] & distcc 2\(\times\), as used in XCode 1.5 and others \\ \hline CVE-2012-1823 [31] & sapi/cgi/cgi\_main.c in PHP \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Group of vulnerabilities. * Exploration rate (\(\epsilon\)): This represents how often the agent tries random actions instead of the one it thinks is best. A higher value means the agent explores more, and a lower value means it sticks with what it knows works. * Decay rate: The rate at which \(\epsilon\) is decayed to favor the exploitation of known actions with a high reward over the exploration of random actions. During the training phase of the Q-Table on the Apache CouchDB vulnerability, we ran seven trials, each running for 500 episodes of exploitation. After training, a vulnerable Apache CouchDB 3.1.0 machine was used to deploy the trained agent. The agent managed to exploit the vulnerable machine in 8.10 s.  
Our approach involved training an AI agent using RL utilizing the Metasploit framework. Metasploit offers a vast collection of payloads for exploiting vulnerabilities in different OSs. The RL algorithm trains the agent to pick the most effective payload for the job. It accomplishes this by using Metasploit's RPC API to automate tasks within the framework. While not all payloads work for every situation, the RL approach helps the agent make smart choices and successfully exploit the vulnerability. Figure 3 shows this process. The following points summarize the training process presented in Figures 2 and 3, given the state of the system (OS and vulnerability):  
1. The agent sends a request to get ta payload to exploit the vulnerability from MSFRPC. 2. The agent chooses a certain payload to use. Here is how the agent decides which payload to use: with probability \(\epsilon\), the agent selects a random payload to be executed (i.e., an action). On the other hand, with probability \((1-\epsilon)\), the agent selects the best known payload. This is an \(\epsilon\)-greedy approach. 3. The agent sends a payload from Metasploit to try and exploit the vulnerability (take action). 4. After using a payload, the agent observes the outcome (new system state) and receives a reward (success or failure signal). This process repeats for a set number of times. As the agent learns, it relies less on random choices (\(\epsilon\) decreases) and focuses on the most successful payloads. Figure 2 demonstrates the complete training process using Q-Tables. Our agent considers a successful exploit to be one that opens a reverse shell session. This might not be the goal for every payload in Metasploit, as some might aim for different types of access (like a VNC session). But, for our purposes, a reverse shell signifies success. To train the agent, we designed a reward system.