Deep Learning and LSTM based Malware Classification | by Ferhat Ozgur Catak | Towards Data Science

Malware development has seen diversity in terms of architecture and features. This advancement in the competencies of malware poses a severe threat and opens new research dimensions in malware detection. This study is focused on metamorphic malware that is the most advanced member of the malware family. It is quite impossible for anti-virus applications using traditional signature-based methods to detect metamorphic malware, which makes it difficult to classify this type of malware accordingly. Recent research literature about malware detection and classification discusses this issue related to malware behaviour.
Cite The Work If you find this implementation useful please cite it:
@article{10.7717/peerj-cs.285,title = {Deep learning based Sequential model for malware analysis using Windows exe API Calls},author = {Catak, Ferhat Ozgur and Yazı, Ahmet Faruk and Elezaj, Ogerta and Ahmed, Javed},year = 2020,month = jul,keywords = {Malware analysis, Sequential models, Network security, Long-short-term memory, Malware dataset},volume = 6,pages = {e285},journal = {PeerJ Computer Science},issn = {2376-5992},url = {https://doi.org/10.7717/peerj-cs.285},doi = {10.7717/peerj-cs.285}}
You can access the dataset from my My GitHub Repository.
Malicious software, commonly known as malware, is any software intentionally designed to cause damage to computer systems and compromise user security. An application or code is considered malware if it secretly acts against the interests of the computer user and performs malicious activities. Malware targets various platforms such as servers, personal computers, mobile phones, and cameras to gain unauthorized access, steal personal data, and disrupt the normal function of the system.
One approach to deal with malware protection problem is by identifying the malicious software and evaluating its behaviour. Usually, this problem is solved through the analysis of malware behaviour. This field closely follows the model of malicious software family, which also reflects the pattern of malicious behaviour. There are very few studies that have demonstrated the methods of classification according to the malware families.
All operating system API calls made to act by any software show the overall direction of this program. Whether this program is a malware or not can be learned by examining these actions in-depth. If it is malware, then what is its malware family. The malware-made operating system API call is a data attribute, and the sequence in which those API calls are generated is also critical to detect the malware family. Performing specific API calls is a particular order that represents a behaviour. One of the deep learning methods LSTM (long-short term memory) has been commonly used in the processing of such time-sequential data.
This research has two main objectives; first, we created a relevant dataset, and then, using this dataset, we did a comparative study using various machine learning to detect and classify malware automatically based on their types.
One of the most important contributions of this work is the new Windows PE Malware API sequence dataset, which contains malware analysis information. There are 7107 malware from different classes in this dataset. The Cuckoo Sandbox application, as explained above, is used to obtain the Windows API call sequences of malicious software, and VirusTotal Service is used to detect the classes of malware.
The following figure illustrates the system architecture used to collect the data and to classify them using LSTM algorithms.
Our system consists of three main parts, data collection, data pre-processing and analyses, and data classification.
The following steps were followed when creating the dataset.
Cuckoo Sandbox application is installed on a computer running Ubuntu Linux distribution. The analysis machine was run as a virtual server to run and analyze malware. The Windows operating system is installed on this server.
We import the usual standard libraries to build an LSTM model to detect the malware.
import pandas as pdimport matplotlib.pyplot as pltimport seaborn as snsfrom sklearn.preprocessing import LabelEncoderfrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import confusion_matrixfrom keras.preprocessing.text import Tokenizerfrom keras.layers import LSTM, Dense, Dropout, Embeddingfrom keras.preprocessing import sequencefrom keras.utils import np_utilsfrom keras.models import Sequentialfrom keras.layers import SpatialDropout1Dfrom mlxtend.plotting import plot_confusion_matrix
In this work, we will use standard our malware dataset to show the results. You can access the dataset from My GitHub Repository. We need to merge the call and the label datasets.
malware_calls_df = pd.read_csv("calls.zip", compression="zip",                               sep="\t", names=["API_Calls"])malware_labels_df = pd.read_csv("types.zip", compression="zip",                                sep="\t", names=["API_Labels"])malware_calls_df["API_Labels"] = malware_labels_df.API_Labelsmalware_calls_df["API_Calls"] = malware_calls_df.API_Calls.apply(lambda x: " ".join(x.split(",")))malware_calls_df["API_Labels"] = malware_calls_df.API_Labels.apply(lambda x: 1 if x == "Virus" else 0)
Let’s analyze the class distribution
sns.countplot(malware_calls_df.API_Labels)plt.xlabel('Labels')plt.title('Class distribution')plt.savefig("class_distribution.png")plt.show()
Now we can create our sequence matrix. In order to build an LSTM model, you need to create a tokenization based sequence matrix as the input dataset
max_words = 800max_len = 100X = malware_calls_df.API_CallsY = malware_calls_df.API_Labels.astype('category').cat.codestok = Tokenizer(num_words=max_words)tok.fit_on_texts(X)print('Found %s unique tokens.' % len(tok.word_index))X = tok.texts_to_sequences(X.values)X = sequence.pad_sequences(X, maxlen=max_len)print('Shape of data tensor:', X.shape)X_train, X_test, Y_train, Y_test = train_test_split(X, Y,                                                    test_size=0.15)le = LabelEncoder()Y_train_enc = le.fit_transform(Y_train)Y_train_enc = np_utils.to_categorical(Y_train_enc)Y_test_enc = le.transform(Y_test)Y_test_enc = np_utils.to_categorical(Y_test_enc)Found 278 unique tokens.Shape of data tensor: (7107, 100)
The LSTM based classification model is then given for example as exercise here:
def malware_model(act_func="softsign"):    model = Sequential()    model.add(Embedding(max_words, 300, input_length=max_len))    model.add(SpatialDropout1D(0.1))    model.add(LSTM(32, dropout=0.1, recurrent_dropout=0.1,                   return_sequences=True, activation=act_func))    model.add(LSTM(32, dropout=0.1, activation=act_func, return_sequences=True))    model.add(LSTM(32, dropout=0.1, activation=act_func))    model.add(Dense(128, activation=act_func))    model.add(Dropout(0.1))    model.add(Dense(256, activation=act_func))    model.add(Dropout(0.1))    model.add(Dense(128, activation=act_func))    model.add(Dropout(0.1))    model.add(Dense(1, name='out_layer', activation="linear"))    return model
The next step is to train the model. I trained and saved my model. Because of the dataset, the training stage takes lots of time. In order to reduce the execution time, you can load my previous trained model from the GitHub repository.
model = malware_model()print(model.summary())model.compile(loss='mse', optimizer="rmsprop",              metrics=['accuracy'])filepath = "lstm-malware-model.hdf5"model.load_weights(filepath)history = model.fit(X_train, Y_train, batch_size=1000, epochs=10,                    validation_data=(X_test, Y_test), verbose=1)Model: "sequential"_________________________________________________________________Layer (type)                 Output Shape              Param #   =================================================================embedding (Embedding)        (None, 100, 300)          240000    _________________________________________________________________spatial_dropout1d (SpatialDr (None, 100, 300)          0         _________________________________________________________________lstm (LSTM)                  (None, 100, 32)           42624     _________________________________________________________________lstm_1 (LSTM)                (None, 100, 32)           8320      _________________________________________________________________lstm_2 (LSTM)                (None, 32)                8320      _________________________________________________________________dense (Dense)                (None, 128)               4224      _________________________________________________________________dropout (Dropout)            (None, 128)               0         _________________________________________________________________dense_1 (Dense)              (None, 256)               33024     _________________________________________________________________dropout_1 (Dropout)          (None, 256)               0         _________________________________________________________________dense_2 (Dense)              (None, 128)               32896     _________________________________________________________________dropout_2 (Dropout)          (None, 128)               0         _________________________________________________________________out_layer (Dense)            (None, 1)                 129       =================================================================Total params: 369,537Trainable params: 369,537Non-trainable params: 0_________________________________________________________________NoneEpoch 1/107/7 [==============================] - 22s 3s/step - loss: 0.0486 - accuracy: 0.9487 - val_loss: 0.0311 - val_accuracy: 0.9672Epoch 2/107/7 [==============================] - 21s 3s/step - loss: 0.0378 - accuracy: 0.9591 - val_loss: 0.0302 - val_accuracy: 0.9672Epoch 3/107/7 [==============================] - 21s 3s/step - loss: 0.0364 - accuracy: 0.9604 - val_loss: 0.0362 - val_accuracy: 0.9625Epoch 4/107/7 [==============================] - 20s 3s/step - loss: 0.0378 - accuracy: 0.9593 - val_loss: 0.0328 - val_accuracy: 0.9616Epoch 5/107/7 [==============================] - 22s 3s/step - loss: 0.0365 - accuracy: 0.9609 - val_loss: 0.0351 - val_accuracy: 0.9606Epoch 6/107/7 [==============================] - 21s 3s/step - loss: 0.0369 - accuracy: 0.9601 - val_loss: 0.0369 - val_accuracy: 0.9606Epoch 7/107/7 [==============================] - 22s 3s/step - loss: 0.0371 - accuracy: 0.9594 - val_loss: 0.0395 - val_accuracy: 0.9625Epoch 8/107/7 [==============================] - 22s 3s/step - loss: 0.0378 - accuracy: 0.9601 - val_loss: 0.0365 - val_accuracy: 0.9588Epoch 9/107/7 [==============================] - 22s 3s/step - loss: 0.0358 - accuracy: 0.9618 - val_loss: 0.0440 - val_accuracy: 0.9456Epoch 10/107/7 [==============================] - 21s 3s/step - loss: 0.0373 - accuracy: 0.9589 - val_loss: 0.0354 - val_accuracy: 0.9644
Now, we have finished the training phase of the LSTM model. We can evaluate our model’s classification performance using the confusion matrix. According to the confusion matrix, the modelâ€TMs classification performance quite good.
y_test_pred = model.predict_classes(X_test)cm = confusion_matrix(Y_test, y_test_pred)plot_confusion_matrix(conf_mat=cm,                      show_absolute=True,                      show_normed=True,                      colorbar=True)plt.savefig("confusion_matrix.png")plt.show()
Let’s continue with the training history of our model.
plt.plot(history.history['accuracy'])plt.plot(history.history['val_accuracy'])plt.title('model accuracy')plt.ylabel('accuracy')plt.xlabel('epoch')plt.legend(['train', 'test'], loc='upper left')plt.grid()plt.savefig("accuracy.png")plt.show()plt.plot(history.history['loss'])plt.plot(history.history['val_loss'])plt.title('model loss')plt.ylabel('loss')plt.xlabel('epoch')plt.legend(['train', 'test'], loc='upper left')plt.grid()plt.savefig("loss.png")plt.show()
The purpose of this study was to create an LSTM based malware detection model using my previous malware dataset. Although our dataset contains instances that belong to some malware families with unbalanced distribution, we have shown that this problem does not affect classification performance.