# Enhancing Malware Detection via RGB Assembly Visualization and Hybrid Deep Learning Models
## 4 Methodology
### Step 1: Data Preprocessing
Integration with automated unpacking tools or hybrid analysis approaches combining static and dynamic methods would be necessary to extend applicability to such challenging samples in real-world scenarios. In response to reviewer feedback, and to empirically validate the limitations of our static analysis approach, we conducted a comparative experiment. The goal was to assess the performance of objdump--the disassembler used in our pipeline--against a state-of-the-art reverse engineering tool on packed executable files. Packed executables are a common obfuscation technique used by malware to hinder static analysis. For this test, we created a small set of Windows PE files and packed them using UPX (v5.0.1), a widely known executable packer. We then analyzed these packed samples with both objdump and Ghidra (v11.3.2), a powerful framework developed by the NSA that includes advanced capabilities for handling obfuscated code. The comparison focused  
Figure 4: PE2 Asm transformation. on whether the disassembler could look past the packing layer to analyze the original program's code, specifically by identifying the Original Entry Point (OEP). As demonstrated in Table 6, objdump was unable to bypass the UPX packing layer. Its output was limited to the disassembly of the packer's stub code, which is responsible for decompressing the original program in memory at runtime. Consequently, the actual logic of the original application remained inaccessible to our analysis pipeline. In contrast, Ghidra successfully identified the presence of UPX, automatically suggested its built-in unpacking script, and allowed for a full analysis of the original, unpacked code. This empirical result provides concrete evidence for the limitation discussed in our manuscript: our current methodology is effective on non-obfuscated or unpacked files but would require pre-processing with dedicated unpacking tools to handle evasive malware samples in real-world scenarios. The example output of objdump typically resembles the following content:  
\"00401000 <_start>:  
401000: 31 ed  xor %ebp,%ebp  
401002: 49 89 d1  mov %erdx,%r9  
401005: 5e  pop %rsi  
401006: 48 89 e2  mov %rsp,%rdx  
401009: 48 83 e4 f0  and $0 xffffffffff0,%rsp\"  
The disassembly output provides critical low-level information for malware analysis, where memory addresses (e.g., 401000) reveal instruction locations and processor operations (e.g., xor, mov) expose malicious functionality. These elements are transformed into a visual representation through a three-channel encoding scheme. The conversion process generates distinct transition matrices capturing different aspects of opcode sequences: the byte transition matrix tracks alphanumeric character patterns across instructions, the opcode transition matrix records initial character sequences, and the quality transition matrix analyzes terminal character relationships. These matrices are mapped to RGB channels (red, green, and blue, respectively), creating composite images that encode the malware's behavioral patterns. The red channel preserves general character-level transitions, while the green and blue channels emphasize opcode-specific structural relationships. This tripartite representation effectively converts sequential assembly instructions into spatial patterns that simultaneously capture local operation sequences and global code structure, serving as distinctive behavioral fingerprints for classification. The resulting visualizations maintain the semantic relationships inherent in the original code while presenting them in a format amenable to deep learning analysis, particularly for convolutional neural networks designed for image processing. \begin{table}
\begin{tabular}{c c c c} \hline \hline
\begin{tabular}{c} Success. Identified UPX packing \\ and enabled analysis of the \\ original program \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Comparative analysis of disassembler performance on UPX-Packed PE files.