# Enhancing Malware Detection via RGB Assembly Visualization and Hybrid Deep Learning Models
## 5 Experimental Setup and Results
* Macro-averaged F1-score is reported alongside micro-averaged metrics to better reflect performance on minority classes. Figure 7: The results for ASM files. Figure 8 presents the model's training and validation performance in terms of accuracy and loss throughout the training process. In the left subplot, the validation accuracy exhibits a rapid increase during the initial epochs, reaching above 90% by epoch 10 and continuing to improve slightly thereafter, suggesting effective generalization. The training accuracy follows a steady upward trend, ultimately stabilizing around 84%, which is slightly lower than the validation accuracy. In the right subplot, both training and validation losses decrease consistently over the epochs. The validation loss shows a smoother and more pronounced decline compared to the training loss, eventually converging to a lower value. The absence of significant divergence between training and validation curves indicates that the model does not suffer from overfitting and maintains strong generalization capability throughout training. Figure 9 illustrates a 2D UMAP (Uniform Manifold Approximation and Projection) visualization of the learned feature representations after classification. Each point corresponds to an individual sample, and the colors represent the predicted class labels across 26 categories. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Precision** & **Recall** & **F1-Score** & **Support** \\ \hline Adposhel & 0.9789 & 1.000 & 0.9894 & 93 \\ \hline Allaple & 1.000 & 1.000 & 1.000 & 88 \\ \hline Amonetize & 1.000 & 0.9885 & 0.9942 & 87 \\ \hline AutoRun & 0.9706 & 0.8684 & 0.9167 & 38 \\ \hline BrowseFox & 0.9744 & 1.000 & 0.9870 & 38 \\ \hline Dinwod & 0.8214 & 0.7931 & 0.8070 & 29 \\ \hline InstallCore & 0.9474 & 0.9890 & 0.9677 & 91 \\ \hline MultiPlug & 0.9794 & 0.9694 & 0.9744 & 98 \\ \hline Other & 0.9274 & 0.9504 & 0.9388 & 121 \\ \hline VBA & 1.000 & 0.9900 & 0.9950 & 100 \\ \hline Vilsel & 1.000 & 0.9744 & 0.9870 & 78 \\ \hline accuracy & & 0.9710 & 861 \\ \hline macro avg & 0.9636 & 0.9567 & 0.9597 & 861 \\ \hline weighted avg & 0.9712 & 0.9710 & 0.9708 & 861 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Comparison of malware families of DumpWare10 dataset by using hybrid CNN model. Figure 8: Graph of Training and Validation Performance Metrics. The plot reveals well-separated clusters for most classes, indicating that the model has successfully learned discriminative features in the latent space. The compactness and distinctness of several clusters suggest that the model can distinguish between many of the classes with high confidence. Some overlap and scattering among certain class representations can be observed, potentially pointing to areas where the model finds it more challenging to differentiate between specific classes. Overall, the UMAP provides qualitative evidence of the model's ability to learn meaningful and structured feature representations aligned with class-wise separability. The model was also tested with the Dumpware10 [6] dataset, which was previously validated. The experimental results of these tests were compared with the MalevisAsm dataset (As shown in Table 7).