# Enhancing Malware Detection via RGB Assembly Visualization and Hybrid Deep Learning Models
## 5 Experimental Setup and Results
### RGB Channel Contribution Analysis  
To validate the effectiveness of our RGB channel separation approach, we conducted comprehensive visual and quantitative analyses using Gradient-weighted Class Activation Mapping (Grad-CAM) [53; 5], which highlights the regions of an input image most \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Method** & **Input** & **Dataset** & **Algorithm** & **Accuracy (\%)** \\ \hline ([PERSON] et al., 2011) [31] & Byte image & \begin{tabular}{c} Malware dataset \\ (unclear) \\ \end{tabular} & Image-based classification & 97.18 \\ \hline [PERSON] et al., 2019 [41] & Byte image & Microsoft BIG 2015 & Image visualization + KNN & 97.40 \\ \hline ([PERSON] et al., 2016) [42] & Byte sequence & VirusShare & Strand & 97.41 \\ \hline ([PERSON] et al., 2016) [43] & Byte image & \begin{tabular}{c} Microsoft Malware \\ Dataset \\ \end{tabular} & Linear kNN & 96.6 \\ \hline ([PERSON] and [PERSON], 2017) [44] & Function call graph & MalGenome & Ensembling multiple RF & 99.3 \\ \hline ([PERSON] and [PERSON], 2022) [45] & Bit sequence & MalwareBazaar & 1D CNN & 96.32 \\ \hline ([PERSON], [PERSON], [PERSON], [PERSON], 2018) [46] & Structural entropy & EMBER & CNN & 98.28 \\ \hline ([PERSON] et al., 2020) [47] & Opcode sequence & Malicia & Self-attention & 98.48 \\ \hline ([PERSON] et al., 2018a) [48] & Byte sequence & Microsoft BIG 2015 & Residual network & 98.61 \\ \hline ([PERSON] et al., 2017) [49] & Byte image & [PERSON] & tGAN & 96.39 \\ \hline ([PERSON] and [PERSON], 2022) [50] & Byte sequence & VirusTotal & VAE + 1D CNN + LSTM & 97.47 \\ \hline ([PERSON] et al., 2019) [51] & Byte image & [PERSON] & CNN & 97.5 \\ \hline ([PERSON] et al., 2020) [52] & Markov image & [PERSON] & VGG16 + SVM & 99.08 \\ \hline ([PERSON] et al., 2023) [37] & \
\begin{tabular}{c} Assembly instruction \\ sequence \\ \end{tabular} & Microsoft Malware & \multirow{2}{*}{CNN} & \multirow{2}{*}{99.44} \\  & sequence & Dataset & \\ \hline \multicolumn{5}{c}{Hybrid CNN Model with} \\ Our Paper & PE2 Asm + Asm2 PNG & Malevis (EXE) &
\begin{tabular}{c} Five-Tuning: \\ EfficientNetB0 \& \\ DenseNet121 \\ \end{tabular} & 98.45 \\ \hline \hline \end{tabular}
\end{table} 
Table 14: Analysis of the proposed model with previous approaches. influential in the model's classification decision. This allowed us to better understand the individual contributions of each RGB channel to malware family identification. For each malware sample, we generated activation maps for the full RGB image, for each individual channel input (with the other channels zeroed), and produced channel-specific Grad-CAM visualizations. We analyzed one representative sample from each of the 26 malware families to qualitatively capture the diversity of malware characteristics across the dataset. Our Grad-CAM analysis reveals distinct activation patterns associated with each RGB channel, highlighting their individual contributions to malware classification. The red channel primarily activates structural boundaries and geometric patterns in the binary visualization, high-entropy regions that often correspond to packed or encrypted code, and distinctive function call or API usage signatures. The green channel shows strong activation in areas containing textural patterns that represent instruction sequences, data section boundaries, and recognizable string or import table structures indicative of dependency relationships. In contrast, the blue channel focuses on contrast variations that signal code-data transitions, background-foreground separation emphasizing critical code regions, and patterns linked to header information or metadata. The accuracy of our proposed model is compared against variants where key components are removed as shown in Table 14. The removal of the RGB-specific branch causes a catastrophic drop in performance, highlighting its critical importance. In the ablation study, it was observed that removing the channel attention mechanism did not affect the model performance (Table 15).