# Q-Learning Approach Applied to Network Security
## 4 Results
### Setting up the Environment as an MDP  
We started by representing the network environment as MDP. We simulated the environment using Python 3.13.3 and predefined the actions, states, rewards, and discount factor. We do not need to define any transition probabilities for the Q-learning algorithm to work. For the agent, the different states will be represented by whether the network is under normal conditions or one of the possible types of attacks. For this example, we will consider five types of network conditions: normal traffic, denial of service (Dos), probe, root-to-local (R2L), and user-to-root (U2R) attacks. The denial of service (DoS) cyberattack occurs when the network becomes unreachable to its legitimate users because the system is overwhelmed with traffic. These attacks can disrupt the normal functioning of online services and applications, resulting in substantial downtime and potential financial losses. Probe attacks are network observation activities in which an unauthorized entity sends distinct packets to our network so they can gather information about our system and possible vulnerabilities. Root-to-local (R2L) attacks are intrusions into our system. An unauthorized entity accesses our network by using our vulnerabilities to gain access and then transmitting packets to a remote machine. User-to-root (U2R) attacks occur when an authorized, nonprivileged user explores the system's vulnerabilities to obtain elevated rights, such as administrator privileges, and take control. * **State space**\(S\) is a countable set of states \(S:\{s_{N},s_{DoS},s_{probe},s_{R2L},s_{U2R}\};s_{N}\) represents the state if the traffic is normal; \(s_{DoS}\) represents the state if the traffic is under denial of service attack; \(s_{probe}\) represents the state if the traffic is under a \(Probe\) attack; \(s_{R2L}\) represents the state if the traffic is under a \(RoottoLocal\) attack; and \(s_{U2R}\) represents the state if the traffic is under a \(UsertoRoot\) attack, \(\forall s\in\mathcal{S}\). * **Action space**\(A:\{a_{block},a_{report},a_{return},a_{allow}\}\), \(a_{block}\) is the action to block the traffic, \(a_{report}\) represents the action to report the traffic to the administrator, \(a_{return}\) is the action to send the traffic back for another classification, and \(a_{allow}\) is to allow the traffic in the system, \(\forall a\in\mathcal{A}\). * **Transition Probability Matrix (\(P\))**: This is the probability of moving from one state to another by taking a particular action; the q-learning agent learns the transition probabilities in finding an optimal policy, which is the state transition probability. Since we have four actions, for each action, we need a \(5\times 5\) matrix, For example, for the action \(block\), the following matrix could be represented as follows: \[P_{ij}^{block}:\left[ \begin{array}{cccc}^{block}{}_{normal,normal}&^{block}{ }_{normal,DoS}&^{block}{}_{normal,Probe}&^{block}{}_{normal,R2L}&^{block}{}_{ normal,U2R}\\ ^{block}{}_{DoS,normal}&^{block}{}_{DoS,DoS}&^{block}{}_{DoS,Probe}&^{block}{} _{DoS,R2L}&^{block}{}_{DoS,U2R}\\ ^{block}{}_{Probe,normal}&^{block}{}_{Probe,DoS}&^{block}{}_{Probe,Probe}&^{block} {}_{Probe,R2L}&^{block}{}_{Probe,U2R}\\ ^{block}{}_{R2L,normal}&^{block}{}_{R2L,DoS}&^{block}{}_{R2L,Probe}&^{block}{} _{R2L,R2L}&^{block}{}_{R2L,U2R}\\ ^{block}{}_{U2R,normal}&^{block}{}_{U2R,DoS}&^{block}{}_{U2R,Pobe}&^{block}{} _{U2R,R2L}&^{block}{}_{U2R,U2R}\end{array} \right]\]
* **Reward (R)**: These are the incentives for good decisions and penalties for mistakes. The reward function helps the agent move in the desired direction, ideally based on a predetermined cost function associated with the agent's costs. The current reward values are selected based on how we would like to train the model and what we ultimately want the agent to achieve, making the agent move in this direction, as the intuition behind reinforcement learning suggests. \[R:\left[