# Enhancing Malware Detection via RGB Assembly Visualization and Hybrid Deep Learning Models
## 4 Methodology  
The methodology consists of three main stages: Data Preprocessing, Convolutional Neural Network (CNN) Design, Classification and Evaluation. For the data preprocessing step, the PE files in the dataset are converted into.asm files. In this model, all.exe files within a folder are processed to generate RGB image output formats. These processes are specifically designed to adapt file formats for different analysis methods, particularly in the context of malware analysis.  
The model processes executables first through asm files. It is dedicated to creating.asm files. Subsequently, the.asm files are converted into RGB image files using separate methods.  
After these processes, the output locations for the bytes and ASM files are determined while maintaining the original folder structure. The necessary folders are created using \"os.makedirs\".  
In the next step, classification is performed using a new model with CNN layers, and the results are evaluated. Additionally, UMAP, a dimensionality reduction and multi-faceted learning technique, is used for this study in the context of malware image transformation problems. The general flowchart is shown in Figure 2.  
Explanations regarding the developed model are given under the following headings: (i) Dataset and Preprocessing  
The dataset consists of malware images generated from assembly code files. The dataset is divided into training, validation, and test sets. The training and validation data are loaded using the image_dataset_from_directory function with an 80/-20 split, while the test dataset is loaded separately. To enhance model generalization, data augmentation techniques such as random flipping, rotation, zoom, and contrast adjustments are applied using TensorFlow's Sequential API.  
(ii) Model Architecture  
A hybrid deep learning model is designed by combining EfficientNetB0 and DenseNet121 architectures. The models are initialized with pre-trained ImageNet weights and are fine-tuned by unfreezing the last 20 layers. The extracted features from both architectures are concatenated and passed through a series of fully connected layers with batch normalization, dropout, and L2 regularization. The final classification layer contains 26 output neurons with a softmax activation function. The model training is performed in a GPU-accelerated TensorFlow environment. Memory growth settings are enabled for efficient GPU utilization. Training and testing times are recorded to analyze computational performance. The model's performance is evaluated using the test dataset. Predictions are obtained in batches for efficient memory usage. The evaluation includes accuracy, confusion matrix visualization, and classification reports generated using scikit-learn's confusion_matrix and classification_report functions.  
(iii) Training Configuration  
Figure 2: The proposed model, which processes images represented by their Red, Green, and Blue (RGB) channels.  
The model is trained using the Adam optimizer with a cosine decay learning rate scheduler. The loss function used is SparseCategoricalKrosentropy, and the evaluation metric is accuracy. Class imbalance is handled by computing class weights using the compute_class_weight function from scikit-learn. The model training includes an early stopping callback to prevent overfitting.  
Training history is visualized through accuracy and loss plots over epochs. Additionally, Uniform Manifold Approximation and Projection (UMAP) is used to visualize high-dimensional feature representations in a 2D space.