## 2 Related Works
### RL  
RL is a form of machine learning that comprises four key components: the agent, the action, the environment, and the reward state. In RL, the agent determines an action based on the reward obtained from the environment. This approach employs rewards and punishments as feedback to encourage desired behavior and discourage undesired behavior, respectively [14]. The model learns from these rewards and punishments through dynamic interactions with the environment. [PERSON] and [PERSON] [15] utilized RL to detect flooding-based distributed denial of service (DDoS) attacks. They devised a simulated network environment, integrating a Q-learning algorithm to manage the injection of anomalies and establish rewards for successful anomaly detection. Nevertheless, a limitation of this research lies in the absence of guaranteed convergence to the optimal solution. The model achieved impressive performance metrics, including an accuracy of 99%, precision of 83%, recall of 100%, and specificity of 100%. [PERSON] et al. [16] proposed a time series anomaly detector using deep reinforcement learning (DRL) that focuses on anomaly-based detection rather than signature-based detection. Their approach aimed to detect anomalies without assuming any underlying mechanism for anomaly patterns, making it adaptable to dynamic environments. The method utilized a logical classifier, which eliminated the need for threshold tuning and simplified the implementation of the RL algorithm. The authors evaluated their approach using accuracy (100%) and recall (100%) metrics on the Yahoo benchmark datasets [17], which served as general time series datasets and were not specifically designed for IDS datasets. The Yahoo benchmark dataset is a generic and scalable anomaly detection framework called EGADS, implemented at Yahoo, which automatically monitors and alerts on large-scale time series data for various use cases. By using these datasets, [PERSON] et al. demonstrated the effectiveness of their DRL-based anomaly detection method in a broader context. Unfortunately, the programming language used for the implementation was not explicitly mentioned in the paper. [PERSON] et al. [18] proposed a framework called Adversarial Environment Reinforcement Learning (AE-RL) for implementing a classifier based on RL theory IDS datasets. The AE-RL framework consists of two agents: the classifier agent and the environment agent. RL is utilized to generate new samples from the environment agent. The main objective of AE-RL is to create an adversarial environment that actively challenges the predictions made by the classifier. In this approach, an intelligent environment is simulated as a second agent, which generates random samples (states) from the training data and assigns rewards based on the classifier's performance with an adversarial objective. The AE-RL framework aimed to improve the classifier's ability to detect and classify anomalies in the IDS datasets. [PERSON] et al. implemented their method using the NSL-KDD IDS dataset and evaluated the performance using various metrics, including time, accuracy, recall, precision, and F1 score. The implementation was conducted in Python version 3.6 on a Windows 10 Pro 64-bit operating system. The authors utilized TensorFlow version 1.15.2 and Keras version 2.2.4 as the main deep learning frameworks for the implementation. It is important to note that the AE-RL framework had a limitation in that it required intelligent sample selection rather than random selection, which could potentially affect its applicability in certain scenarios. In comparison to previous methods, the paper by [PERSON] et al. demonstrated superior performance with an F1 score of 0.78, accuracy of 0.7744, precision of 0.7616, and recall of 0.7744. The results demonstrated the effectiveness of the AE-RL framework in improving the detection performance of IDS systems. [PERSON] et al. [3] proposed an adversarial RL framework with SMOTE (AESMOTE) for anomaly detection.