# Enhancing Malware Detection via RGB Assembly Visualization and Hybrid Deep Learning Models
## 4 Methodology
### Step 1: Data Preprocessing
The assembly files are processed with automatically determined encoding, and the opcode (operation) and operand (parameter) information are extracted. Following this, transition probability matrices for byte, opcode, and quality levels are calculated based on character transitions and then normalized. Three different transition matrices are combined as RGB channels to generate an image. The images are saved in 8-bit RGB format as.png files. With parallel processing, multiple files are processed simultaneously, reducing the processing time, and the status of the tasks is regularly reported. As a result, the RGB images are saved in the specified directory while maintaining the class-based folder structure. The encoding of the.asm files is automatically determined, ensuring that the files are correctly processed regardless of their initial format. Once the encoding is identified, information about the opcode (operation) and operand (parameter) is extracted from the assembly code. Transition probability matrices are calculated for byte, opcode, and quality levels based on character transitions within the assembly instructions. These matrices are then normalized to ensure consistent values across the datasets. Afterward, the three transition matrices are combined into RGB channels, resulting in the creation of an image that visually represents the opcode patterns. The generated images are saved in 8-bit RGB format as.png files, which ensures efficient storage while maintaining image quality. To enhance processing efficiency, multiple files are handled simultaneously through parallel processing, reducing the overall computation time. The status of each processing task is continuously reported to monitor the progress. The resulting RGB images are saved in the specified directory, organized in a manner that maintains the class-based folder structure. This organization allows for easy access and further analysis of the images. (ii) ASM to Image Converter (Asm2 Image):  
This study presents a systematic approach for converting executable byte files into RGB-format PNG images to facilitate the visual analysis of assembly code. MCTVD method was used for Asm2 Image transformation [37]. The proposed conversion pipeline comprises several key stages, including assembly file preprocessing, transition matrix computation, image generation, parallel processing, and structured output storage. As shown in Figure 5, the conversion process involves the following steps:  
(i) Assembly File Processing  
The system begins by automatically detecting the character encoding of.asm files to ensure accurate text parsing and prevent potential misinterpretation of assembly instructions. This crucial preprocessing step accounts for various encoding formats that may be encountered in different disassembly outputs. Following successful encoding validation, the pipeline performs comprehensive instruction extraction, isolating both opcodes (machine instructions) and their corresponding operands (parameters) from the assembly code. This extraction process meticulously preserves the syntactic structure and semantic relationships within the code, maintaining the original program logic while preparing the data for subsequent analysis stages. The extracted opcode-operand pairs form a complete representation of the executable's functionality, capturing not just individual operations but their contextual relationships and execution patterns. This dual-level preservation proves particularly valuable for malware analysis, as it enables the reconstruction of both low-level processor commands and higher-level program behavior. The system handles various assembly syntax variations and mnemonics while maintaining consistent output formatting for analysis. Error-checking mechanisms verify the integrity of extracted instructions, flagging any malformed or ambiguous code segments for further inspection. This robust preprocessing ensures that subsequent visualization and classification stages operate on accurate, standardized representations of the original binary's behavior.