## 4 The Proposed Methods
In order to convert malware samples into binary images, a malware binary is taken as an 8-bit unsigned integer vector and afterwards arranged into a two-dimensional array. Despite significant variety, the visual textures of the same families share a lot of similarities [36]. Nonetheless, the MaleVis dataset's RGB byte images of the PE binary files are visible. Therefore, these RGB images need to be converted to grayscale. Figure 10 illustrates how images of malware that are RGB have been transformed into images of malware that are grayscale. Subsequently, the resizing method will modify the malware images' dimensions to fit the given input size for every pretrained CNN model. The new malware images' dimensions are 224 \(\times\) 224 or 227 \(\times\) 227. ### 2.2 Feature Extraction

Malware classification techniques rely primarily on the feature extraction procedure. Because Pretrained models have already learned to obtain appropriate features from large amounts of data, they are employed as feature extractors for malware image detection. In this step, several models ranging in complexity and layer number are used, including ResNet50, GoogLeNet, AlexNet, DenseNet201, and SqueezeNet. These models will be used on our datasets in order to reduce the overfitting issue and determine which model is most effective at classifying images as malware. This step procedure is the input image that will be processed across the pretrained model, and the image's feature vector is obtained by using the output of the final layer before the layers are eliminated. By using transfer learning, we can train deep learning models on small datasets without overfitting. ### 2.3 ResNet50 Features Extraction

ResNet-50 is only utilized for feature extraction in this case. Instead of being used for classification, the network is used as a feature extractor. This indicates that the extracted feature vector is the result of the final pooling layer before the classification layer. The basic structure of ResNet50 for feature extraction is displayed in the Algorithm 2. The ResNet50 feature vector that is produced is \(1\times 2048\). Figure 10: The graphical representation of the transformation operation [4]. Step 2.2: DenseNet201 Features Extraction

The network architecture of DenseNet-201 consists of several dense blocks, transition blocks, and an ultimate classification layer that will be eliminated. The purpose of the transition blocks is to minimize the feature maps' spatial dimensions in the areas between dense blocks. The basic structure of DenseNet201 for feature extraction is displayed in the Algorithm 3. The DenseNet201 feature vector that is produced is 1 \(\times\) 1000 from the last fully connected layers (fc1000). ```
Input: RGB Malware Image from dataset. Output: \(1\times 1000\) features vector dimension. Begin For  1: Upload the preparation malware image. 2: Normalize each image input that falls between the 0 and 255. 3: Change the resolution to (224 \(\times\) 244). 4: Load the ResNet201 model. 5: Load DenseNet201 model weights. 6: The model's last fully connected (FC) layers have been eliminated or are not included. 7: Build an entirely new model for feature extraction. 8: Using ResNet50's final convolutional layer (\"avg_pool\" or \"pool5\"), the activation is produced. 9: An array has been generated from the image. 10: Reducing the features array's dimensions to one. 11: Print the flattened array. End for End
```
**Algorithm 2** ResNet50 model as feature extraction. ```
Input: RGB Malware Image from dataset. Output: \(1\times 1000\) features vector dimension. Begin For  1: Upload the preparation malware image. 2: Normalize each image input that falls between the 0 and 255. 3: Change the resolution to (224 \(\times\) 244). 4: Load the DenseNet201 model. 5: Load DenseNet201 model weights. 6: The model's last fully connected (FC) layers have been eliminated or are not included. 7: Build an entirely new model for feature extraction. 8: Using DenseNet201 final convolutional layer (\"fc1000\"), the activation is produced.