# Enhancing Malware Detection via RGB Assembly Visualization and Hybrid Deep Learning Models
## 5 Experimental Setup and Results
### Results of Previous Work on Malevis Dataset  
With high precision and robust performance, the model effectively classifies the 25 malware families in the MaleVis dataset. Overfitting was mitigated by capping training at 20 epochs, and approaches like bicubic interpolation improved both generalization and training efficiency. Furthermore, fine-tuning and class weight balancing helped manage the dataset's imbalance. The experimental results demonstrate that our hybrid CNN model achieves superior performance compared to existing malware classification approaches, attaining 98.45% accuracy on the benchmark dataset. As shown in Table 12, this represents a significant improvement over previous methods, including a 4.85% increase over MobileNet fine-tuning (96.04%) and a 2.96% enhancement compared to the best prior CNN approach (95.82% F1-score). Several key factors contribute to this performance advantage. First, the assembly based visualization approach (Asm) proves more effective than byte-level representations, as evidenced by our model's 5.35% higher accuracy compared to the best byte-based method (93.10%). This suggests that preserving semantic relationships through \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Metric** & **Adam** & **AdaBoB** & **AdaBelief** \\ \hline Top-1 Accuracy & 0.9845 & 0.9984 & 0.9990 \\ \hline Weighted F1-Score & 0.9845 & 0.9978 & 0.9990 \\ \hline Weighted Precision & 0.9848 & 0.9981 & 0.9991 \\ \hline Weighted Recall & 0.9845 & 0.9984 & 0.9990 \\ \hline Training Time (s) & 1329.74 & 1513.41 & 1522.34 \\ \hline Best Validation Loss & 0.0155 & 0.0016 & 0.0001 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Optimizer performance evaluation. opcode sequences provides more discriminative features than raw byte patterns. Second, the hybrid architecture combining EfficientNetB0 and DenseNet121 extracts complementary features that capture both local texture patterns and global structural relationships, addressing the limitations of single-architecture approaches. The results reveal interesting patterns across methodologies. Traditional machine learning techniques (Random Forest, SVM) plateau around 92-94% accuracy, while deep learning approaches consistently surpass 95%. Notably, the performance gap widens substantially (6.82% in F1-score) when comparing our method to the best traditional approach (SMO-RBF), underscoring the advantages of hierarchical feature learning for malware classification. Our model shows particular strength in recall (99.48%), indicating exceptional capability in identifying positive malware samples. This characteristic is crucial for security applications where false negatives carry significant risk. The precision-recall balance (97.45-99.48%) suggests the visualization method effectively minimizes false positives while maintaining thorough detection coverage. As shown in Table 13, our model requires 1329.74 s for training and 47.75 s for inference, using only 8 GB of GPU memory. Compared to [PERSON] et al. [6] which achieves faster inference (3.56 s) on a 24 GB CPU, our method is more memory-efficient on GPU. [PERSON] et al. [40] report faster training (1056 s) but provide no inference data. [PERSON] and [PERSON] [39] used 12 GB GPU memory but did not report timing results. Overall, our model offers a balanced trade-off between speed and resource usage. The experimental results demonstrate that our hybrid CNN model achieves superior performance compared to existing malware classification approaches, with a notable accu \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Method** & **Training Time (s)** & **Inference Latency (s)** &
\begin{tabular}{l} **GPU/CPU Memory** \\ **GB** \\ \end{tabular} \\ \hline Our Model & 1329.74 & 47.75 & 8 GB GPU \\ \hline [PERSON] et al. (2021) [6] & - & 3.56 & 24 GB CPU \\ \hline [PERSON] and [PERSON] (2022) [39] & - & - & 12 GB GPU \\ \hline [PERSON] et al. (2024) [40] & 1056.0 & - & 16 GB GPU \\ \hline \hline \end{tabular}
\end{table} 
Table 13: Performance comparison of our model and prior work.