## 3 Proposed Visualized Malware Multi-Classification Framework
### Fine-Tuning and Classification Phase  
In general, TL can be performed in three different ways [50]: (1) shallow tuning: the last layer in the model is assigned to a new task, and the constraints of the other model's layers are frozen, (2) deep tuning: the whole end-to-end parameters of the pre-trained CNN architecture are retrained, and (3) fine-tuning: the CNN layers are gradually trained by fine-tuning the learning hyperparameters till a remarkable performance enhancement is accomplished. This paper applied fine-tuning, which is a compromised approach between the other two tuning types. Fine-tuning takes advantage of efficient classification tasks in the case of imbalanced datasets. Additionally, it causes lower complexity than the deep tuning and better classification accuracy than the shallow tuning. The employed eight CNN models previously trained on the ImageNet dataset that contains 1000 different classes are adapted to our malware detection challenge. This adaptation is implemented through fine-tuning their layers' parameters and weights. In these models, the output layer that contains 1000 classes is modified and fine-tuned to comprise 25 classes (25 malware families). Additionally, as will be discussed, the original weights of the primary pre-trained CNN models of the ImageNet dataset were initially used, and after that, they were optimized and fine-tuned based on the back-propagation technique [51]. The fine-tuning process of the layers' weights is an iterative optimization procedure that is performed and repeated until determining the best value of the filter weights (\(w\)) that achieve a minimum error rate. The used cost function is expressed in Equation (1)  
\[F(w,K)=\frac{1}{m}\sum_{j=1}^{m}f\big{(}p\big{(}k_{j},w\big{)},\hat{c}_{j}\big{)} \tag{1}\]  
where \(m\) refers to the malware images contained in the training dataset \(K\), \(p(k_{j},w)\) is the CNN prediction function that predicts the class \(c_{j}\) of \(k_{j}\) by assuming the value of \(w\), \(\hat{c_{j}}\) is the proper class of the \(j^{th}\) malware image, \(k_{j}\) is the \(j^{th}\) malware image of \(K\), and \(f(c_{j},c_{j})\) is the logistic error function that predicts \(c_{j}\) rather than \(\hat{c_{j}}\). During the fine-tuning process, the performance efficiency of three different optimizers [52]: (1) RMSprop (Root Mean Square Propagation), (2) SGDM (Stochastic Gradient Descent with Momentum), and (3) ADAM (Adaptive Moment Estimation Optimizer) were examined to select the best optimizer for estimating the optimal filter weights of the CNN layers. As a result, the ADAM optimizer was used for finding the optimal \(w\) due to its superior performance compared to other optimizers. ADAM optimizer combines the main advantages and benefits of the other two SGDM and RMSprop optimizers, where it establishes adaptive learning rates for each parameter in the training process. Hence, the significant improvement of the ADAM optimizer is that it retains an exponentially decaying average of the past squared gradient descent to reach a minimum value faster. Further details, mathematical expressions, and descriptions of the ADAM optimizer can be found in [53]. In the training process, a massive capacity of hardware memory is required to store the filter weights \(w\) of the CNN layers, so a mini-batch size is set to 64. The learning rate is regularly altered until the optimal value is reached; where high learning rates cause overfitting, while slow learning rates limit the error variants among epochs. Therefore, the initial learning rate was set to 0.00001; to efficiently regulate the update of the weight sizes. The CNN layers' weights were updated in each iteration, and the mini-batches were iterated for every epoch. The max-epoch was set to 20, where this value was selected through observing validation errors throughout fine-tuning process using various learning rates.