# Enhancing Malware Detection via RGB Assembly Visualization and Hybrid Deep Learning Models
## 5 Experimental Setup and Results  
After completing the training phase, the total training duration is calculated and displayed. During the testing phase, predictions are generated by processing the test dataset in smaller batches, and the total test duration is also measured and printed. The model then predicts class labels based on the test data, obtaining the final predicted values by selecting the class with the highest probability. Our baseline selection prioritized architectures with established performance in RGB malware image classification scenarios, providing a robust foundation for evaluating our hybrid approach across diverse architectural paradigms in cybersecurity contexts. All experiments were implemented in the Python (3.12.4) programming language on the Ubuntu (22.04.3 LST) operating system and executed using a GPU. The TensorFlow library (2.18.0) with the Keras framework (3.6.0) was utilized to run the models. The system specifications used in the experiments consisted of NVIDIA GeForce RTX 3070 Laptop GPU graphics card with 8 GB memory, 16 GB RAM and AMD Ryzen 9 5900 HX with Radeon Graphics. To analyze the classification performance, the true labels of the test dataset are extracted and compared against the predicted labels. A confusion matrix is computed to visualize classification errors and overall prediction performance. The matrix is plotted using a color-coded representation, making it easier to interpret class-wise accuracy and misclassifications. The model's performance over epochs is analyzed through visualizations. The accuracy trends for both training and validation data are plotted, highlighting how well the model learns over time. Similarly, the loss function is visualized to observe potential overfitting or underfitting issues. To gain deeper insights into the feature representation, the predictions are transformed into a lower-dimensional space using the UMAP technique. The resulting two-dimensional embeddings are plotted, where data points are color-coded according to their true labels. This provides an intuitive view of how well the model separates different classes. Finally, a classification report is generated, presenting precision, recall, and F1-score for each class. This detailed breakdown helps assess the model's performance across different categories, identifying strengths and potential areas for improvement. As evidenced by the confusion matrix in Figure 7, the model exhibits varying classification accuracy across different classes. Notably, in the balanced MaleVis dataset, merely 11 out of the 25 malware families were correctly identified. Our analysis revealed suboptimal classification for viral strains: Sality (97%) and Agent (86%). While these performance metrics remain reasonably high, six additional malware families showed accuracy levels near 90%, collectively contributing to the model's overall accuracy reduction to 98.45%. The model architecture demonstrates competitive performance across all evaluation metrics, achieving 97.45% precision, 99.48% recall, and a 98.5% F1-score, confirming its effectiveness for the task. The experimental results empirically validate the efficacy of our image-based approach for malware classification, demonstrating its practical utility in this domain. Despite class imbalance challenges, our approach achieved competitive results. Table 5 shows per-class precision and recall values, demonstrating that even minority classes (with <100 samples) achieved reasonable performance. The weighted average F1-score of 98.5% indicates robust performance across all classes. In the experiments, it was observed that there were promising results despite class imbalance. Given the inherent class imbalance in the dataset, we conducted a comprehensive analysis of its impact on model performance:  
* Per-class performance metrics are reported in Tables 7 and 8. * Confusion matrix analysis (Figure 7) reveals which minority classes are most affected. * We employed weighted loss functions to mitigate the imbalance effect.