# Q-Learning Approach Applied to Network Security
## 4 Results
### Episode Reward and Q Table  
Figure 2 visualizes the training progression of a Q-learning agent over 300 episodes. It shows how the reward changes as the agent learns the optimal policy. The blue line represents the reward obtained in each episode, while the red line represents the moving average reward. From episodes 0 to 25, the agent starts with low rewards as it has no prior knowledge of the environment. As we progress from episode 25 to 75, the agent begins to learn an effective policy, and there is a steady increase in reward. The moving average also increases consistently, indicating that the agent is discovering better decision-making strategies for intrusion detection. From episode 75, the rewards stabilize, indicating that the agent converges towards an optimal policy. The moving average reward curve flattens around episode 100, indicating that the agent has successfully learned to differentiate the different states and has selected the best action based on learned experiences. Hundred episodes in terms of network security are executed almost instantly without any delay. Table 2 shows the Q-Table generated after training. It represents the action-value function, the cumulative reward for each state-action pair, guiding the agent in its decision-making. The values in the table represent the long-term benefit of choosing a particular action in a state; higher values indicate more favorable conditions. Based on those Q-values, which represent the optimal solution to the Bellman Equation (3), the agent obtains an optimal policy, and for each state, he selects the best possible action. For example, if we consider the first row of the table above, we will observe that the state the agent is currently in is \"Normal\" and the highest Q-value of that row is associated with the action \"Allow\". Therefore, the agent will prefer to \"Allow\" the traffic if he is aware that \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**State** & **Allow** & **Report** & **Return** & **Block** \\ \hline Normal & 5.57542983 & 0.77936266 & 1.58838193 & \(-\)0.34863757 \\ DoS & \(-\)4.10741098 & 3.92943439 & 0.69046204 & 10.88718444 \\ Probe & 0.76225638 & 1.76297612 & 2.79851319 & 5.71924872 \\ R2L & \(-\)1.19494085 & 1.66870395 & 4.69743104 & 6.74777392 \\ U2R & \(-\)2.24912002 & 0.72623961 & 3.84500612 & 8.68402438 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Q-table. Figure 2: Episode reward for Q-learning agent. the current state is \"Normal\". The agent always needs to know its initial state in order for the Q-learning algorithm to work. However, he doesn't need to estimate the transitional probabilities for the optimal solution to be provided. Therefore, Q-learning is a very suitable approach for providing optimal responses to the environment. The learning process is based on trial and error in the beginning, until the best policy is obtained. The action with the highest Q-value maximises the Q-value function and is an automotive response that could be adopted in the IR system. This outcome creates an optimal policy for the agent. This policy may or may not be unique. The next row is if the agent is under a \"Dos\" attack, one of the very challenging attacks for the system. The best possible action for this state, as suggested by the Q-learning algorithm, is the \"Block\" action. However, sometimes the best course of action is to \"Report\" the traffic to the administrator, because a human interaction may be preferred, especially for malicious traffic that is isolated and under \"quarantine\". Continuing further, the next states in the table indicate that the best possible action is to \"Block\" the traffic, depending on the severity of the attack. The sensitivity analysis of the different levels of gamma \(\gamma\) (0.1, 0.3, 0.5), as well as the levels of iterations of the model, are represented in Figure 3. We plotted the moving average reward across 1000 training episodes. With a \(\gamma\) value of 0.1, the model converges more quickly and stably than with 0.3 and 0.5.