# Enhancing Malware Detection via RGB Assembly Visualization and Hybrid Deep Learning Models
## 5 Experimental Setup and Results
\begin{table}
\begin{tabular}{c c c c c} \hline  & **Precision** & **Recall** & **F1-Score** & **Support** \\ \hline Adposhel & 0.9300 & 1.000 & 0.9637 & 100 \\ \hline Agent & 0.9400 & 1.000 & 0.9691 & 91 \\ \hline Allaple & 0.4500 & 1.000 & 0.6207 & 96 \\ \hline Amonetize & 0.9700 & 0.8800 & 0.9229 & 100 \\ \hline Androm & 1.000 & 0.8300 & 0.9071 & 100 \\ \hline Autorun & 0.9900 & 1.000 & 0.9950 & 99 \\ \hline BrowseFox & 1.000 & 0.9900 & 0.9950 & 99 \\ \hline Dinwod & 0.6000 & 1.000 & 0.7500 & 25 \\ \hline Elex & 0.9400 & 0.9900 & 0.9644 & 100 \\ \hline Expiro & 1.000 & 1.000 & 1.000 & 100 \\ \hline Fasong & 1.000 & 1.000 & 1.000 & 100 \\ \hline HackKMS & 1.000 & 1.000 & 1.000 & 100 \\ \hline Hlux & 1.000 & 0.9900 & 0.9950 & 100 \\ \hline Injector & 1.000 & 1.000 & 1.000 & 93 \\ \hline InstallCore & 0.5000 & 1.000 & 0.6667 & 2 \\ \hline MultiPlug & 1.000 & 1.000 & 1.000 & 100 \\ \hline Neoreklami & 1.000 & 1.000 & 1.000 & 100 \\ \hline Neshta & 1.000 & 1.000 & 1.000 & 100 \\ \hline Regrun & 0.8300 & 1.000 & 0.9071 & 5 \\ \hline Sality & 0.9900 & 0.9800 & 0.9850 & 98 \\ \hline Snarasite & 1.000 & 1.000 & 1.000 & 100 \\ \hline Stantinko & 1.000 & 1.000 & 1.000 & 100 \\ \hline VBA & 1.000 & 1.000 & 1.000 & 100 \\ \hline VBKrypt & 0.9800 & 1.000 & 0.9899 & 48 \\ \hline Vilsel & 1.000 & 1.000 & 1.000 & 100 \\ \hline benign & 1.000 & 0.8600 & 0.9247 & 965 \\ \hline accuracy & & & 0.9430 & 3121 \\ \hline macro avg & 0.9315 & 0.9777 & 0.9426 & 3121 \\ \hline weighted avg & 0.9657 & 0.9430 & 0.9501 & 3121 \\ \hline \end{tabular}
\end{table} 
Table 8: Comparison of malware families of MalevisAsm dataset by using gray images. Table 8 shows the results obtained by converting the MalevisAsm RGB dataset to gray color. According to this table, it is observed that testing the images as RGB produces higher accuracy. In Table 9, the performance of the model developed for malware classification was tested using RGB (color) images. In order to prove that RGB images provide better performance, tests were also performed with grayscale images (Table 9). The findings clearly show that the model trained with RGB images exhibits a clear performance superiority over the model trained with grayscale images. While the RGB model achieved a high overall accuracy rate of 98.45%, the accuracy of the grayscale model remained at 94.30%. This performance increase of approximately 4.15% shows that color information is a critical feature for the discrimination capacity of the model. The process of converting images to grayscale reduces the three-channel (Red, Green, Blue) color information to a single-intensity channel. This process causes the loss of distinctive information that arises during visualization from the binary structures of malware and corresponds to certain color tones or patterns. Since the model cannot feed on these lost color features, it has difficulty distinguishing classes from each other. To validate the robustness of our proposed method and ensure that our findings are not contingent on a specific optimization algorithm, we conducted an ablation study on the choice of optimizer. In addition to the widely used Adam optimizer employed in our main experiments, we evaluated our model's performance with two advanced optimizers: AdaBelief and AdaBoB. AdaBelief adapts the step size by considering the \"belief\" in the current gradient direction, while AdaBoB [38] integrates this mechanism with the dynamic learning rate bounds from AdaBound. For a fair comparison, all optimizers were configured with the same initial learning rate and experimental setup. The results are presented in Table 10. The findings clearly indicate that our method is not only robust, but its performance can be significantly enhanced by  
Figure 9: UMAP Representations for MalevisAsm dataset. more advanced optimizers. Both AdaBoB and AdaBelief substantially outperformed Adam across all evaluation metrics. Notably, Top-1 Accuracy increased from 98.45% with Adam to 99.84% with AdaBoB and 99.90% with AdaBelief.