Malware Detection Using Deep Learning | by Ria Kulshrestha | Towards Data Science

Malware refers to malicious software perpetrators dispatch to infect individual computers or an entire organization’s network. It exploits target system vulnerabilities, such as a bug in legitimate software (e.g., a browser or web application plugin) that can be hijacked.
A malware infiltration can be disastrous — consequences include data theft, extortion or the crippling of network systems.
Malware is one of the most serious security threats on the Internet today. In fact, most Internet problems such as spam e-mails and denial of service attacks have malware as their underlying cause. That is, computers that are compromised with malware are often networked together to form botnets, and many attacks are launched using these malicious, attacker-controlled networks.
In order to deal with the new malware generated, new techniques to detect them and prevent any damage caused by them.
Nowadays deep learning has dominated the various computer vision tasks. Not only these deep learning techniques enabled rapid progress in this competition, but even surpassing human performance in many of them. One of these tasks is Image Classification.
Unlike more traditional methods of machine learning techniques, deep learning classifiers are trained through feature learning rather than task-specific algorithms. What this means is that the machine will learn patterns in the images that it is presented with rather than requiring the human operator to define the patterns that the machine should look for in the image. In short, it can automatically extract features and classify data into various classes.
Early layers learn how to detect low-level features like edges, and subsequent layers combine features from earlier layers into a more holistic and complete representation.
We can transform a malware/benign file into a grayscale image using the method described later. Then we can apply these deep learning techniques on the generated images to classify them as malware or benign.
To classify images using a deep learning model we will need images from both benign and malware files. We will only do a binary classification (malware and benign class). Multi-class classification can also be done using this technique, with the idea being that a variant of malware files will have images different from the other.
Once we have our dataset ready, we will convert each file into a 256x256 grayscale image (each pixel has a value between 0 and 255) by doing the following steps for each image:
Step 1: Read 8 bits at a time from the file.Step 2: Treat the 8 bits as a binary number and convert it to its    corresponding integer.Step 3: Enter the number as the pixel value.
A file of maximum size 64 KB can fit a 256 x 256 image. Any file having a size greater than 64 KB, its remaining content will be dropped. On the other hand, if the size of a file is less than 64 KB the remaining image will be padded with 0’s.
Since malware detection is done in real time, we need to classify an image as benign or malware within seconds. Therefore, keeping the image generation process simple and short will help us save valuable time.
This step is extremely simple. Once all the images have been generated segregate them into two folders — train and validation. Each of these folders will contain two folders namely — Malware and Benign.Store these two folders in another folder named “dataset.tar” and compress it to generate a .tar file.
So the final directory structure will be →dataset.tar contains Validation and Train. Validation will have folders Malware and Benign. Train will have folders Malware and Benign.
Move the compressed folder in the same directory as the Jupyter notebook containing the code.
** The model used for this problem is created by following the lesson 1 of the fast.ai course(Pet classification). **
To read the dataset from your Google Drive add the following lines of code
!pip install PyDrive #import osfrom pydrive.auth import GoogleAuthfrom pydrive.drive import GoogleDrivefrom google.colab import authfrom oauth2client.client import GoogleCredentialsauth.authenticate_user()gauth=GoogleAuth()gauth.credentials = GoogleCredentials.get_application_default()drive = GoogleDrive(gauth)
Once you run the last 4 lines of the above-mentioned code, Google SDK will ask you to enter a verification code. It will mention a link, once you click on it you’ll get your security code.
link = '1sL4I4xNh657AhrIOOwbr6TX58ahyC'(add the link here )
Create a variable which contains the link to the .tar file containing the data which is stored in your Google Drive. To get the URL open the .tar file in your drive and get the sharable link to the file. You don’t need the entire URL only a part of it. So for example, if your sharable link is “https://drive.google.com/File/d/1sL4I4xNh657AhrIOOwbr6TX58ahyC/view?usp=Sharing”you only need the part which is in bold.
downloaded = drive.CreateFile({'id':link}) downloaded.GetContentFile('dataset.tar')import tarfiletar = tarfile.open("dataset.tar")path = tar.extractall('DS/')tar.close()path = Path.cwd().joinpath("DS/dataset/")path.ls()data = ImageDataBunch.from_folder(path, ds_tfms=get_transforms(), size=224)## To view data in a batchdata.show_batch(rows=3, figsize=(7,6))## To know the number of classes in a datasetprint(data.c)## To know the names of the classes in a datasetprint( data.classes)
learn = create_cnn(data, model.resnet34, metrics = error_rate)## To know the model architecturelearn.model## Training the modellearn.fit_one_cycle(4)learn.fit_one_cycle(10)
Here is a screenshot of the output I got after running the above code:
interp = ClassificationInterpretation.from_learner(learn)losses,idxs =  interp.top_losses()interp.plot_top_losses(9,figsize=(9,6))
So far we haven’t told our model what learning rate to use while training the model, which we all know is one of the most important hyper-parameter while training.In order to find a good learning rate, do the following:
learn.lr_find()learn.recorder.plot()
learn.fit_one_cycle(5, max_lr=slice(1e-6,1e-4))## Saving the weights of the modellearn.save('stage-1-malware-detection')
Note: Every time you call the fit_one_cycle() function, the weights aren’t re-initialized. So if you call the function one after the other, the epochs add up, meaning if you call fit_one_cycle(5) followed by fit_one_cycle(10) the model has been trained for 15 epochs approx.
The complete code can be found on my GitHub account which is linked here. These are just a few snippets of the code and just following them won’t give a workable code.
Please feel free to reach out for any suggestions/improvements/discussions on the same. :D
Instead of performing a binary classification we can try classifying the data into different types of malware or benign class.
We can try different techniques to create a validation set. At this stage, the images in different classes, as well as the number of images in the validation set and train set, were chosen at random.
Try different ratios of the number of malware files to the number of benign files in our training dataset. (Almost 1:1 used)
Try different dimensions to generate malware images. (256x256 used)
Yes, you should listen to Andrej Karpathy, and understand Back propagation
Evaluation of an NLP model — latest benchmarks
Machine Learning in Computational Fluid Dynamics
I‘m glad you made it till the end of this article. 🎉I hope your reading experience was as enriching as the one I had writing this.💖
Do check out my other articles here.
If you want to reach out to me, my medium of choice would be Twitter.