# Q-Learning Approach Applied to Network Security
## 2 Literature Review
They also estimate each state to reduce computational power. The so proposed method can be successfully applied to IDS. In another paper [5] [PERSON] proposes another anomaly detection approach for sequential data founded on TD learning, where a Markov reward function is presented. The author claims that TD in reinforcement learning can successfully detect abnormal behavior in the case of elaborated sequential processes by estimating the value of the Markov reward function. The advantage of the suggested model is that there is a straightforward labelling procedure utilizing delayed signals. The accuracy can be improved even with a limited training set, and it is superior to the accuracy obtained with support vector machines (SVM) and HMMs. The performance metrics of the presented anomaly detection procedure using TD learning are estimated from a system call data of host-based ID from the MIT Lincoln Lab and the University of New Mexico (UNM). The author suggests a sequential anomaly detection approach for multistage attacks based on temporal-difference (TD) learning. A Markov reward model is created, and it is also demonstrated that the value function in the Markov reward model is analogous to the anomaly probability of the data sequences. The off-policy methods encounter policies that optimize the action-value function during learning by utilizing unused theoretical actions. [PERSON], [PERSON], [PERSON], and [PERSON] [6] suggest a model incorporating a Q-learning algorithm and rough set theory (RST) for IDS. The algorithm aims to achieve the highest classification accuracy by categorizing the NSL-KDD data set as \"normal\" or \"anomaly\". The data are discretized by applying cut operation attributes. The Q-learning algorithm is adjusted so the decision agent can learn the optimum cut value for different attributes to attain higher accuracy. The authors present two phases of the reward: the initial and final reward matrix. The proposed algorithm lowers some of the complexity of the Q-learning and obtains 98% accuracy. The authors in [7] employ a pursuit reinforcement competitive learning (PRCL) approach [8; 9] for ID. This method applies immediate reinforcement learning, where feedback is obtained at each phase after making a decision. It executes clustering in real-time with high accuracy in detecting intrusions. The proposed system consists of data preprocessing, the PRCL algorithm, and the performance evaluation phase. Three methods related to competitive clustering RL are compared, and the accuracy is recorded. [PERSON], [PERSON], and [PERSON] [10] applied RL to detect Botnet using PRCL with further rule detection, which has reward and penalty rules to achieve a solution. Based on the empirical result, PRCL can detect Botnet accurately in real-time. PRCL uses an unsupervised dataset to cluster the Botnetand obtain accuracy; it can achieve clustering online. [PERSON] and [PERSON] [11] suggest a wireless sensor networks (WSN) detection algorithm assuming a Q-learning method on a hybrid IDS. The authors simulated twenty sensors, WSNs, which communicate through the dynamic source routing protocol for hierarchical representation networks. The tested sensor nodes are clustered in four regions of 100 m \(\times\) 100 m. Twenty sensors were selected, and the Q-learning algorithm was applied. The result outperforms other machine learning methods based on accuracy, precision, and false favorable rates. Some of the more recent authors, including [12; 13] represent the network as a combination of more agents and provide a broader application of the reinforcement learning aspect.