# Enhancing Malware Detection via RGB Assembly Visualization and Hybrid Deep Learning Models
## 2 Related Work
### Memory-Based Analysis  
Memory analysis has gained prominence as it bypasses the limitations of static analysis when facing obfuscated disk data. However, these approaches encounter substantial challenges including the ephemeral nature of memory evidence that can be quickly overwritten or cleared [6]. Advanced malware employs memory evasion techniques such as process hollowing, reflective DLL loading, and fileless attacks that minimize memory footprints, making detection difficult. Unlike disk-based methods, memory retains unencrypted execution artifacts (DLLs, processes, network activities) until system shutdown, enabling the detection of evasive malware. This approach requires no unpacking and operates OS independently, though specialized hardware tools are often needed for reliable acquisition. The studies on memory-based analysis in the literature are summarized in Table 3.  
We contribute to malware detection research in four significant ways:  
(1) Novel Assembly-to-RGB Visualization Technique: We introduce a novel Assembly-to-RGB visualization technique that encodes opcode transitions as three-channel matrices, preserving structural semantics lost in grayscale representations. Unlike prior work [PERSON] et al. [31], our mapping explicitly separates opcodes (red), operands (green), and control flow (blue) into distinct channels. This approach directly addresses the signature-based detection limitations by creating visual representations that remain consistent despite code modifications, while capturing semantic relationships invisible to traditional byte-sequence analysis. (2) Hybrid CNN Architecture: We propose a hybrid CNN architecture combining EfficientNetB0 (for local pattern detection) and DenseNet121 (for global feature reuse). This \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Authors** & **Model** & **Advantage** & **Accuracy (\%)** & **Dataset** \\ \hline \multirow{3}{*}{[PERSON] et al. [25]} & \multirow{3}{*}{Enceders + Attention layer} & On altered test samples, it has a high detection rate and surpasses previous models. & 99.98 & Ki dataset \\  & & & Malware detection using graphs generated from API calls & 99 & API call dataset \\ \hline \multirow{3}{*}{[PERSON] et al. [28]} & LSTM-Adam & The best binary & 96.44 & \multirow{3}{*}{Catak dataset} \\  & LSTM-RMSProp & classification model is & 97.30 & \\ \cline{1-1}  & GRU-Adam & achieved using LSTM & 96.62 & \\ \cline{1-1}  & GRU-RMSProp & and the RMSProp & 96.44 & \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Summary of dynamic analysis studies. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Authors** & **Model** & **Advantage** & **Accuracy (\%)** & **Dataset** \\ \hline \multirow{3}{*}{[PERSON] et al. [29]} & \multirow{3}{*}{BiLSTM SPP-Net} & Detect and classify IoT malware & \multirow{3}{*}{92.5} & Korea Internet \& Security Agency (KISA) \\  & & & & (KISA) \\ \hline \multirow{3}{*}{[PERSON] et al. [30]} & \multirow{3}{*}{BiLSTM} & Safeguard IoT devices against & \multirow{3}{*}{94.46} & Korea Internet \& Security Agency (KISA) \\  & EfficientNetB3 & obfuscated malware in a smart city. & & (KISA) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Summary of Memory-Based Analysis Studies. dual-architecture approach overcomes the feature extraction limitations of single-model systems and provides robust detection capabilities against evasion techniques through complementary feature learning mechanisms. Recent advances in CNN-based malware detection convert binaries into grayscale images for classification, leveraging architectures like VGG or ResNet (Table 1). While these methods outperform traditional approaches, they face three key limitations: (1) Single-architecture designs struggle to generalize across diverse malware families; (2) Feature extraction lacks multi-scale diversity, hindering detection of novel pattern combinations; and (3) Grayscale representations lose critical semantics (e.g., opcode-operand relationships), impairing behavioral analysis (Table 1).