# Enhancing Malware Detection via RGB Assembly Visualization and Hybrid Deep Learning Models
## 4 Methodology
### Step 2: Classification  
Convolutional Neural Networks (CNN) are deep learning algorithms commonly used in image processing, which take images as input. This algorithm, consisting of various layers, captures features from the images using different operations and classifies them accordingly. The image passes through layers such as Convolutional, Pooling, and Fully Connected. The Convolutional layer is the first layer in CNN algorithms that processes the image. As is known, images are essentially matrices composed of pixels containing specific values. In this layer, a smaller filter, compared to the original image size, slides across the image and attempts to capture certain features from it. The Pooling layer reduces the input size, which in turn decreases the number of parameters within this activation. In the Fully Connected layer, the image, which has passed multiple times through convolutional and pooling layers and is in matrix form, is flattened into a vector. Our proposed model, illustrated in Figure 6, employs a hybrid architecture that synergistically combines two powerful pre-trained convolutional neural networks (CNNs), EfficientNet-B0 and DenseNet-121, to act as parallel feature extractors. This dual-backbone design is motivated by the hypothesis that each network can capture complementary visual patterns from our RGB-encoded malware representations. Both backbones were initialized with weights pre-trained on ImageNet. To adapt these models to our specific domain, we applied a fine-tuning strategy where the majority of the layers were kept frozen, while only the final 20 layers of each network were unfrozen and made trainable. This preserves the rich, general-purpose features learned from ImageNet while allowing the model to learn domain-specific features relevant to malware classification. The feature extraction process for each backbone can be formulated as follows:  
\[f\_eff=\varphi_EfficientNet(x) \tag{1}\]  
Figure 6: Architecture of the CNN used in the study. \[f\_dense=\varphi\_DenseNet(x) \tag{2}\]  
where  
* \(x\) is the input RGB image of size \(224\times 224\). * \(\varphi\_EfficientNet(x)\) and \(\varphi\_DenseNet(x)\) represent the feature extraction functions of the EfficientNet-B0 and DenseNet-121 backbones, respectively. * \(f\_eff\in\mathbb{R}^{1280}\) is the resulting feature vector from the EfficientNet-B0 branch after Global Average Pooling. * \(f\_dense\in\mathbb{R}^{1024}\) is the resulting feature vector from the DenseNet-121 branch after Global Average Pooling. The two feature vectors are then fused using a concatenation operation to create a unified, high-dimensional feature representation:  
\[f\_combined=\left[f\_eff\bigoplus f\_dense\right] \tag{3}\]  
where \(\oplus\) denotes the concatenation operator. The dimension of the combined feature vector \(f\_combined\) is \(1280+1024=2304\). Following feature fusion, the combined feature vector is passed through a classification head designed to map the features to the final class probabilities while preventing overfitting. The head consists of the following sequence of layers:  
1. A Batch Normalization layer to stabilize activations and improve gradient flow. 2. A Dense (Fully Connected) layer with 256 neurons and a ReLU activation function. This layer is responsible for learning the high-level discriminative patterns from the fused features. 3. An L2 Regularization term (with \(\lambda=0.01\)) is applied to the dense layer to penalize large weights and reduce model complexity. 4. A Dropout layer with a high rate (\(p=0.6\)) to randomly deactivate neurons during training, further mitigating overfitting. 5. A final Batch Normalization layer to maintain stability before the output layer. 6. A Softmax Output Layer with 26 neurons, corresponding to the number of malware families plus the benign class. To address the significant class imbalance present in the dataset, we employ a Weighted Cross-Entropy loss function.