## 3 Proposed Methodology
### Attack Classification
#### 3.2.2 Enhanced Deep Reinforcement Learning
In the proposed EDQL model, deep AE can be utilized to proficiently approximate the Q-function instead of a Q-table. The selection of deep AE is mainly driven by proficiently handling the high-dimensional and continuous nature of network traffic data. The network traffic data renders conventional Q-tables unfeasible owing to their inability to generalize and the exponential increase in size. In order to detect new attacks, deep AEs are excellent at compressing complex input data into lower-dimensional representations. This progresses generalization and learning stability. Deep AE helps lessen the overestimation bias frequently connected to Q-learning by offering more precise Q-value approximations depending on learned features instead of discrete values. Also, by facilitating fast learning through experience replay and breaking correlations in training data, the deep AE can be simply combined with other unsuper  
Figure 3: Semi-supervised workflow. vised approaches for improving intrusion detection ability without necessitating a lot of manual labeling. Q-tables, shallow neural networks, and linear function approximators are some other techniques used for estimating the Q-function. Owing to their size and lack of generalization, Q-tables are not viable for high-dimensional spaces. Shallow networks, as well as linear function approximators, have issues in capturing non-linear and complex relationships in data, which results in under-fitting. In contrast to other prevailing models, the deep AE is better for approximating the Q-function due to its overall scalability, flexibility, and robustness. Also, the feature extraction and representation learning ability of deep AE makes them more effective for Q-value estimation in complex environments such as attack detection. The state representation can be the input of deep AE. This can be a high-dimensional vector that resembles the current state of the environment. In the proposed method, the deep AE model combines multiple AE layers over one another with connections from one layer to the subsequent layer. In this connection, the output from a first encoder will be the input to a second encoder, where higher feature representations can be learned. The DAE model utilized in the proposed work for approximating the Q-function is displayed in Figure 4.  
An AE is a kind of unsupervised neural network that uses backpropagation to learn from unlabeled data. For an accurate reconstruction of the state in the decoder part, the model learns the following hypothesis function:  
\[H_{w,\beta}(x)\approx x \tag{11}\]  
where \(w\) is the weight value, \(\beta\) is the bias value and x is the input value of a model. The AE includes one encoder and one decoder, where the encoder is responsible for converting the input to low-dimensional representation, and the decoder is responsible for reconstructing the input. The process of encoding reduces the dimensionality of a data space by converting the input into a latent space representation. This can be achieved by imposing a constraint on the hidden neurons of the encoder model to compress the input. The constraint can be formulated as follows:  
\[\delta_{j}=\varrho \tag{12}\]  
Figure 4: Deep AE model used for approximating the Q-function. \[\delta_{j}=\frac{1}{p}\sum_{i=1}^{p}[\alpha_{j}(x)] \tag{13}\]  
where \(\delta_{j}\) is the average activation, \(q\) is the sparsity parameter set close to 0, the \(p\) is the number of training instances, and \(\alpha_{j}(x)\) is an activation of the hidden neuron. If the activation value is 0, then the neuron is inactive; if the activation value is 1, then the neuron is considered active. The mean squared error (MSE) is chosen as the cost function of this network model, which can be mathematically expressed as follows. \[MSE=\frac{1}{p}\sum_{i=1}^{p}\frac{1}{2}||H_{\omega,\beta}(x^{i})-(y^{i})||^{2} \tag{14}\]  
Along with this loss function, the L2 regularization or weight decay term is added to minimize the overfitting issues.