## 4 Proposal
### Modeling and Training  
This section explores the design and training of different machine learning models oriented toward Anomaly Intrusion Detection Systems. selecting four different versions of the observed models in the related works: Decision Trees (DT), random Forest (RF), XGBoost (XGB), and Multi-layer Perceptron (MLP); as well as the proposed Reinforcement Learning model. For each architecture, six models have been trained over each of the reduced feature datasets presented above both with and without balancing. Public implementations have been used when available, opting for the default configuration varying instead the dataset preparation as stated above. The Scikit-learn implementations were used for the first two, the XGBoost library was used for the algorithm with the same name, and Keras was used for MLP. For MLP, a network architecture of two dense layers of 64 units was used, which is the default configuration of the neural network used in the model. Furthermore, EarlyStopping was applied with the following hyper-parameters: monitor = 'val_loss', patience = 10, and restore_best_weights = True, ensuring that the model would stop training if the validation loss did not improve for 10 consecutive epochs and the best-performing model was retained. While the training was set to a maximum of 100 epochs, the model typically executed around 50 to 60 epochs before early stopping was triggered  
Reinforcement Learning Model  
Due to the philosophy behind reinforcement learning models, it is necessary to first define the environment in which the agent was trained. The configuration of the observation space and action space was considered as follows:  
* **Observation Space**\((s)\): of class Box with shape \((x)\), where \(x\) is the number of features of the corresponding training partition. In this way, the agent only observes one record \begin{table}
\begin{tabular}{c c c} \hline \hline
**Threshold** & **Total Number of Features** & **Selected Features** \\ \hline \hline \multirow{3}{*}{0.03} & \multirow{3}{*}{10} & \multicolumn{2}{c}{sbytes, service, smean, ct\_srv\_dst, ct\_srv\_src,} \\  & & ct\_src\_dport\_ltm, ct\_dst\_src\_ltm, ct\_dst\_sport\_ltm, \\ \cline{1-1}  & & ct\_dst\_ltm, dbytes \\ \hline \hline
0.02 & 14 & sload, proto, sttl, dmean \\ \hline \multirow{3}{*}{0.01} & \multirow{3}{*}{28} & ct\_src\_ltm, rate, dur, sinpkt, sjit, dload, dloss, spkts, \\  & & dinpkt, synack, dpkts, djit, sloss, tcprtt \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Description of features included in each dataset. Each set also considers the features of the higher thresholds. at a time. It is also defined that the values provided are of type float within the limits \([-1,1]\). * **Action Space**\((a)\): of class Discrete with nine possible values, each corresponding to a type of attack. For the design of the environment logic, it has been necessary to consider the following points:
* An RL algorithm essentially seeks the solution to a problem modeled, where a state transition depends on the current state and the action taken [1]. * A subset of records from the original dataset is used, where none of the elements have a temporal relationship with the previous one. * The choice of an action (classification of a record) has no effect on the next observation or its features. With this in mind, the environment \(R(\tau)\) is designed such that an episode \((\tau)\) consists of only one state-action pair, so that:  
\[\tau=(s_{0},a_{0})\] \[R(\tau)=r_{t=0}\]  
Otherwise, the RL algorithm would attempt to learn correspondences in transitions from one record to another, slowing down-training or, in the worst case, reducing the model performance by adding a component that could be considered noise. The choice of algorithm was limited by three factors:  
1. Availability in Stable-Baselines3: The library includes six algorithms (A2C, DDPG, DQN, PPO, SAC, and TD3). 2.