## 3 Proposed Methodology
### Attack Classification
#### 3.2.2 Enhanced Deep Reinforcement Learning  
In this section, the proposed EDQL model for attack classification is discussed. The base of the EDQL model is deep Q-network (DQN) theory, a free-environment interaction model in which the state of the succeeding phase is offered directly by the dataset rather than by interaction with the environment. A new training sample is produced during each training iteration by taking samples from the initial training sample, which encompasses the state \(t_{u}\) at the time \(u\), the action \(b_{u}\) at time \(u\), and the set of every action \(B=\{b_{0},b_{1},b_{2},\ldots,b_{p-1},b_{p}\}\) and the state \(t_{u+1}\) at the next time. Initially, the set of actions \(B\) and the state \(t_{u}\) are input into the neural network at time \(u\) to predict every possible Q-value functions \(q(t_{u},B)=\{q(t_{u},b_{0}),(t_{u},b_{1}),\ldots,(t_{u},b_{p-1}),(t_{u},b_{p})\}\) The maximum Q-value function \(maxq_{\pi}(t_{u},B)\) is filtered with the policy algorithm \(\pi\) for state \(t_{u}\). The IFSO is further employed in choosing the actions. The architecture of enhanced deep reinforcing learning is offered in Figure 2.  
The reward function is made up of \(0/1\) rewards that are projected to be correct or incorrect. The return or reward is computed when relating the actual action value \(b_{u}\) under state \(t_{u}\) to the above-predicted action value \(b_{u}^{*}\). The reward is 1 if the two are equal; otherwise, the reward is 0. The aim of the subsequent phase is to predict the Q-value function by means of state \(t_{u+1}\) at time \(u+1\). An unsupervised learning technique is employed to predict the feature labels of the test samples, henceforth improving the ability of the model to identify attacks. This is because the test samples in the dataset encompass normal and attack samples. Thereby, the proposed method first predicts the action to which they belong by means of the deep autoencoder (AE) model. The model is initially trained on the labeled data to establish a baseline understanding of attack  
Figure 2: Architecture of enhanced deep reinforcement learning. and normal classification. The deep AE processes the unlabeled data for constructing feature representations, which are then employed together with labeled data to refine the Q-function and enhance policy selection through IFSA. As the model encounters new data during training, it can adaptively query for labels on instances and include new labeled data into the training set for repeated enhancement. The flowchart of the semi-supervised workflow is provided in Figure 3. The process starts with raw network traffic data, which is separated into unlabeled and labeled datasets. Initially, the labeled data are employed to train the model, whereas unlabeled data are processed through deep AE to extract the features. Then, these extracted features are utilized to approximate the Q-function and tend to optimal policy selection by IFSA for classifying traffic as attack or normal. The final outcome is the classification output, which can be further refined by continual retraining as new data become available. Deep Autoencoder  
Deep AE is a type of neural network architecture that is employed for dimensionality reduction and unsupervised learning applications. Deep AE is made up of an encoder and a decoder in which the encoder utilizes compression to transform the input data into a lower-dimensional representation represented as latent space, and the decoder employs this representation to reconstruct the original input data [32]. Approximating Q-functions in an RL environment is one of the uses for deep AE. Q-functions in RL are applied to define the expected future reward of executing a specific action in a given state. The Q-values for each state-action grouping are stored in a Q-table in conventional RL approaches such as Q-learning. Nevertheless, owing to the vast amount of alternative states, applying a Q-table becomes unrealistic for situations with continuous or huge state spaces.