# Q-Learning Approach Applied to Network Security
## 2 Literature Review  
Intrusion detection methods are widespread in the network security literature. Their goal is to detect attacks and deviations in network traffic. Traditional approaches are based on comparing the evaluated traffic with known signatures. This approach is a misuse detection method unsuitable for encountering unseen attacks [1]. The anomaly detection methods do not need any previous details of particular signatures and can find new attacks. However, they deal with a lot of noise and uncertainty and, therefore, have elevated false-positive rates. A preliminary element of the IDS is an element received from a sensor at a given time; this could be either a system call in-host-based IDSs or a feature in network-based IDSs. A series of anomaly detection is a time-series sequence of features classified as normal or abnormal. In [2], [PERSON] proposes a modified reinforcement learning method for employing adaptive neural networks to ID. The author analyzes the denial-of-service (DOS) attacks employing a cerebellar model articulation controller (CMAC) neural network ([PERSON], 1975). The first stage starts with training the CMAC with simulated attacks, followed by a single test iteration. Three hundred DoS attacks are tested, and the least mean square learning is used to update the weights of CMAC. The reported error was 3.24%. In the second stage, the adaptive learning capabilities of the model are tested to identify differences from DoS attacks. The CMAC is evaluated based on the initial performance before and after receiving feedback from the environment. Before receiving input from the environment, the reported model error is 15.2%, and after that, it improves to 0.4%. Based on the results, the author claims that the CMAC model can obtain meaningful and rapid learning for known and unknown attacks and perform online adaptive learning with significant accuracy. Another category is based on the reinforcement learning policy, for which the action-value function is maximized. A policy is the proposed action that the agent takes, conditional on each state. The RL process could be categorized into on-policy and off-policy learning methods. On-policy methods use a known general policy, and after an action is selected, the model revises the value functions. After exploring and obtaining environmental feedback to optimize the action-value function, the policy may be modified. [PERSON] and [PERSON] [3] propose a kernel-based RL method for sequential behavior modeling in-host-based IDSs, using system call sequences. They consider a kernel-induced feature space and least-squares temporal difference (LS-TD) algorithm. The model could be introduced as a sequential prediction case, which is resolved by utilizing reward signals. The authors empirically present excellent performance metrics of their proposed model compared to the Hidden Markov Models (HMMs) and linear TD algorithms. In order to reduce the feature space, a kernel approach is applied, where a high-dimensional nonlinear feature mapping can be created by selecting a Mercer kernel function in a reproducing kernel Hilbert space (RKHS). The kernel-based LS-TD learning algorithm produces dimensions equal to the number of state transition samples. [PERSON], [PERSON], and [PERSON] [4] suggest an IDS model based on temporal-difference-based sequence anomaly detection 2 (TDSAD2). Their model differs from the classical temporal model in terms of how the transitional probabilities are estimated. The classical methods require knowing the probability distribution for the transitional probabilities. The transition probability is updated by matrix A, representing the pair transitions on dependency on previous states and the number of single states. The authors propose the primary adjustment related to estimating transitional probabilities, overcoming the weakness of previous TD approaches. They keep estimating the pair transition probabilities instead of the occurrence number and introduce the dependence on the previous entrance of observed states.