## 3 ENN
### Architectures of Models
#### 3.3.3 Deep Neural Network (DNN)  
The DNN model has been designed to handle both binary and multi-class classification tasks. It starts with an input block, consisting of a dense layer with ReLU activation, where the input dimension matches the number of features in the dataset. This layer introduces non-linearity, allowing the model to capture complex patterns in the data. The hidden layers are divided into two blocks. The first hidden block includes a dropout layer to prevent overfitting, followed by a dense layer with ReLU activation and then batch normalization to improve training stability and speed. The second hidden block mirrors this structure, adding another dropout layer and batch normalization. The output block is customized based on the classification task. For binary classification, it includes a dense layer with a single neuron and a sigmoid activation function to compute probabilities for the two classes. For multi-class classification, it features a dense layer with a number of neurons equal to the target classes, along with a softmax activation function to predict probabilities for each class. The model was compiled using the Adam optimizer, configured with an ExponentialDecay learning rate schedule to gradually reduce the learning rate during training. Binary cross-entropy was used as the loss function for binary classification, and categorical cross-entropy for multi-class classification. Accuracy served as the evaluation metric in both scenarios. In a DNN, the feed-forward process involves transmitting the input through successive layers to generate the final output. This operation for layer 1 can be mathematically represented as shown in Equation (20) [54]. \[a^{(I)}=f\left(W^{(I)}\,a^{(I-1)}+b^{(I)}\right) \tag{20}\]  
where \(a^{(I)}\) represents the activation of the current layer 1, the weight matrix for this layer is denoted by \(W^{(I)}\), and \(b^{(I)}\) stands for the bias vector of layer 1. The activation function f is applied element-wise and can include non-linear functions such as ReLU or sigmoid, enabling the model to capture complex relationships. The ReLU activation function presented in Equation (21) [55] is a straightforward and effective non-linear function. It outputs zero for any negative input values, encourages sparsity in activations, and facilitates smooth gradient flow, which makes it highly suitable for deep learning models. \[ReLU(x)=max(0,\,x) \tag{21}\]  
For binary classification tasks, the sigmoid function is employed to produce a probability score that reflects the likelihood of an instance belonging to the positive class. It converts the raw score into a value between 0 and 1, acting as a threshold for making the classification decision. This process is mathematically expressed in Equation (22) [53]. \[\sigma(Z)=\frac{1}{1+e^{-z}} \tag{22}\]  
where \(Z\) represents the output from the final dense layer. For multi-class classification tasks, the output layer uses the softmax function, allowing the model to generate probability distributions over multiple classes. This function transforms a vector of raw scores (logits) into normalized values within the range of 0 to 1, ensuring that the sum of all probabilities equals 1. The mathematical representation of the softmax function is provided in Equation (23) [53]. \[Softmax(Z_{i})=\frac{e^{z_{i}}}{\sum\limits_{j}e^{z_{i}}} \tag{23}\]  
where \(Z_{i}\) represents the raw score for the i-th class. (i) Binary Classification  
The architecture outlined in Table 18 describes the structure of a DNN model tailored for binary classification using both the CICIDS2017 and NF-BoT-IoT-v2 datasets. The input layer processes the features of the datasets, with the CICIDS2017 dataset having 69 features and the NF-BoT-IoT-v2 dataset having 18 features. The input layer includes a dense layer with 1024 neurons, activated by ReLU, introducing non-linearity and enabling complex decision boundaries.