# Deep Learning Methods for Malware and Intrusion Detection: A Systematic Literature Review

Authors: Rahman Ali; Asmat Ali; Farkhund Iqbal; Mohammed Hussain; Farhan Ullah
Journal: Security and Communication Networks
 (2022)

---

## Abstract

Android and Windows are the predominant operating systems used in mobile environment and personal computers and it is expected that their use will rise during the next decade. Malware is one of the main threats faced by these platforms as well as Internet of Things (IoT) environment and the web. With time, these threats are becoming more and more sophisticated and detecting them using traditional machine learning techniques is a hard task. Several research studies have shown that deep learning methods achieve better accuracy comparatively and can learn to efficiently detect and classify new malware samples. In this paper, we present a systematic literature review of the recent studies that focused on intrusion and malware detection and their classification in various environments using deep learning techniques. We searched five well-known digital libraries and collected a total of 107 papers that were published in scholarly journals or preprints. We carefully read the selected literature and critically analyze it to find out which types of threats and what platform the researchers are targeting and how accurately the deep learning-based systems can detect new security threats. This survey will have a positive impact on the learning capabilities of beginners who are interested in starting their research in the area of malware detection using deep learning methods. From the detailed critical analysis, it is identified that CNN, LSTM, DBN, and autoencoders are the most frequently used deep learning methods that have effectively been used in various application scenarios.


---

## Body

## 1. Introduction

Deep learning (DL) is a representation learning approach having multiple levels of representation, each of which transforms the representation at one level to a representation at a higher level allowing very complex functions to be learned. Deep learning methods may help in solving the problems that have been a challenge for the machine learning community. It has been found very useful in discovering complex structures in high-dimensional data and is applied in various domains of government, business, and science. Representation learning methods allow us to feed a machine with raw data and discover the representation we need for classification or detection [1, 2]. Machine learning is used for enhancing the functionality of computer systems and data processing systems in various fields, like medicine, research, robotics, web search engines, content filtering, and recommendation systems on social networks and e-commerce platforms, and in products, like cameras and smartphones. However, traditional machine learning techniques have certain limitations in processing data in raw form [2].Malware is a piece of code written with the intent to cause harm to, or subvert, the functionality of a computer system. It is a general term that describes viruses, spyware, Trojans, adware, and other types of malicious code [1]. A malware detecting system is a system that tries to determine whether or not a specific program or piece of code is malicious [1, 3]. Malware developers use obfuscation methods to change the form of malware and deceive virus scanners that use pattern matching techniques because they are purely semantic and ignore the instructions’ semantics and the pattern matching techniques are not resilient to variations [3]. Malware designed today are polymorphic and metamorphic, having the ability to change their code with propagation. Malware variants share various behavioral patterns that could be exploited to detect unknown malware using machine learning techniques that could not be achieved by using the traditional malware detection techniques [4]. Deep learning techniques have been widely used in various fields, including computer vision, speech recognition, pattern recognition, NLP, and malware detection and classification, and have become an active area of research. It is a challenging task to develop systems that can detect any kind of malicious code accurately in a short time. Due to the increasing number and variants of malware, a malware detection system should automatically perform without or with minimal human intervention. Moreover, signature-based malware detection techniques are not sufficient to fight against malware as they could be easily deceived in an intelligent manner [5]. Using deep learning methods for malware detection and classification enables us to build scalable models that may handle any measure of data and improve their accuracy as they can identify more features than traditional machine learning methods. After the training phase, DL models can acquire a new pattern of malware easily [6].From the extensive literature survey of the deep learning methods in the area of malware and intrusion detection, one can understand that researchers have worked in this direction; however, the majority of studies focus on deep learning-based methods or they are related to a specific type of malware (e.g., android malware detection or Windows-based malware detection or network anomaly detection, etc.). Very few surveys can be found, such as [7–9], discussing the subject area; however, they reviewed a very limited number of research works and techniques.A separate study for each type of malware could be conducted, such as Windows-based malware, Android-based malware, intrusion, network anomalies, and other threats to security. Moreover, it is also a good idea to conduct an SLR for different types of threats, such as ransomware; however, we feel the need to present a broad picture and provide a broad-scoped vision to the new researchers in the area, separating the different platforms by discussing them in different sections.A large amount of data is being produced online and in various organizations, which makes this era the era of big data. Traditional data processing and traditional machine learning methods may not be able to process this data and extract insights from it. Deep learning is thought to be more capable of processing large volumes of data and enable the researchers to reach better solutions. During the recent years, researchers have been focusing on deep learning methods for malware and threats detection and classification. Keeping in view the effectiveness and importance of deep learning techniques and the recent trends in research, we conducted a systematic literature review (SLR) of DL techniques used in malware and intrusion detection systems in the last six years, 2015 to 2022.The aim of this study is to identify the scope, trends, and methods of deep learning algorithms exploited in the area of malware detection systems for various platforms and develop a better understanding of the new researchers in this area. To achieve the said aim, the following objectives are set as research questions.(i)
RQ1: Which platforms are affected the most by various types of malware attacks?(ii)
RQ2: What are different malware analytics (hot era of the research for malware detection, most vulnerable platform and most popular journals publishing malware detection literature)?(iii)
RQ3: Which methods are used by the research community for malware detection?(iv)
RQ4: What are the major DL algorithms used in the domain of malware detection?(v)
RQ5: What are the main challenges that we face in using deep learning methods for malware and intrusion detection?(vi)
RQ6: Do the proposed android malware detection systems, approaches, and studies support the characteristics of sustainability, evolvability, and automatically picking the most appropriate malware detection DL algorithm?(vii)
RQ7: Do the Android-based surveyed methods have the ability of identifying any newly introduced malware and what type of feature analysis technique (s) the researchers have used in their research?(viii)
RQ8: What are the most commonly used datasets used for malware detection in the Android-based and Windows-based environments?The rest of the paper is structured as follows. In Section2, a summary of the surveys done in the subject area is made. In Section 3, the research methodology is elaborated from different perspectives. In this section, we summarize the selected studies and present a complete picture of how DL methods are used for malware detection in Windows and Android platforms. In Section 4, experiments are performed to show how DL methods supersede traditional ML methods. Section 5 critically discusses different aspects of the study. Section 6 recommends key points to the readers and those who are new to the area. Finally, Section 7 provides a conclusion of the study.


## 2. Existing Surveys on Deep Learning Methods for Malware Detection

There have been a number of relevant studies in the domain of malware and intrusion detection systems using machine learning, deep learning, or other methods. Some of these studies are summarized in Table1.Table 1
State-of-the-art research in deep learning for malware detection.
Ref.YearDescriptionLimitations[10]2019Survey of approaches that detect permissions demanded by apps that might be used for malicious activities(i) Limited to android malware detection(ii) Permission-based malware detection only[11]2017Deep learning techniques to detect network anomalies(i) Only network anomaly and intrusion detection systems[12]2019Survey of approaches to network anomaly detection(i) Only network anomaly detection(ii) Traditional learning based[13]2018Different malware detection techniques, like signature- and behavior-based detection(i) Not limited to deep learning[14]2018A survey of intrusion detection techniques in vehicular ad hoc networks(i) Limited to intrusion detection systems in vehicular networks[6]2018A survey of android malware analysis using deep learning with static analysis, dynamic analysis, and hybrid analysis(i) Limited to android malware detection(ii) Reviewed very few publications[15]2019A survey of machine learning techniques used in cyber security, like spam detection, phishing detection, and malware detection(i) Traditional learning-based systems(ii) Limited to cyber security[7]2019Review of deep learning-based android malware detection techniques(i) Limited to android malware detection(ii) Reviewed very few publications[16]2019A review of different intrusion detection systems in IoT, including anomaly based, specification based, signature based, and hybrid IDS’s(i) IoT-based systems only(ii) Not limited to DL- and ML-based methods[17]2018A survey of IDS’s and defense systems in IoT(i) IoT-based systems only(ii) Not limited to DL and ML[5]2018A survey of applications of deep learning techniques in malware detection(i) Reviewed only the different DL algorithms used for malware detection[18]2020A survey of deep learning techniques in defense against phishing(i) Limited to phishing attacks only[19]2019A survey of android malware detection systems(i) Limited to Android malware(ii) Not DL or ML based[20]2019A survey of techniques for security in IoT(i) Only IoT-based systems(ii) Not limited to DL[21]2019A survey of intrusion detection systems using deep learning(i) Only intrusion detection systems[8]2020A review of malware detection systems using deep learning(i) Reviewed only about 35 publications[22]2020An extensive review of malware detection systems based on deep learning(i) Limited to Android malware[9]2020A review of DL algorithms used for malware detection and some relevant literature(i) Reviewed very few publicationsProposed studyDeep learning methods for malware and intrusion detection—a systematic literature reviewKey contributions(i) One of the extensive surveys covering a large number of research articles (94) in Windows-, Android-, and IoT-based environments for malware and intrusion detection using deep learning approaches.(ii) Extraction of deeper malware analytics(iii) Extraction of useful information about deep learning methods applied in the domain of malware and intrusion detection(iv) Identification of the most effective deep learning algorithms for malware and intrusion detection(v) Highlighting the key challenges faced during the use of deep learning methods for malware and intrusion detectionIt is evident from the last column of Table1 that these surveys are related to malware or intrusion detection systems; however, most of them are not deep learning-based or related to a specific type of malware (e.g., android malware detection or network anomaly detection). Very few surveys were found that reviewed malware detection systems using deep learning, such as [7–9]; however, they reviewed a very limited number of research works and techniques. In the proposed study, we have tried our level best to cover the use of deep learning algorithms applied to Windows-based, Android-based, and IoT-based environments for the detection and classification of deep learning algorithms. This will provide a base to the beginners in the area to start their research work and easily find the gap for further improvements and start of new research.Key contributions of the survey are enlisted as follows:(i)
One of the extensive surveys covering a large number of research articles (94) in Windows-, Android-, and IoT-based environments for malware and intrusion detection using deep learning approaches(ii)
Extraction and formulation of malware analytics from the relevant literature on DL methods used for malware and intrusion detection(iii)
An extensive study of the relevant literature to extract useful information about deep learning methods used in the domain of malware and intrusion detection systems(iv)
An analysis of which deep learning algorithms are used in malware and intrusion detection systems(v)
Highlighting the key challenges faced by the research community in using deep learning methods for malware and intrusion detection


## 3. Research Methodology

To assess the applications and impact of deep learning methods for malware and intrusion detection systems, a systematic literature review (SLR) is conducted. In an SRL, a systematic approach is followed to identify and analyze relevant studies regarding a specific area of interest [23]. There are a number of guidelines defined by experts for conducting a comprehensive and effective SLR. We studied and followed the guidelines provided in [24, 25].


### 3.1. Research Design

Firstly, we do need an assessment to conduct the literature review. We studied a number of papers from various sources, including journals and conferences. We observed that deep learning has been used by a large number of researchers in malware detection, intrusion detection, and other security-related systems. Based on the need assessment, we formulated a number of research questions, mentioned in the introduction section, to show the effectiveness and outcomes of the proposed study.We then selected a number of search libraries to make sure the selected literature is from an authentic source and is reliable. We selected five different libraries, including Google Scholar, Science Direct, IEEE Explore, ACM, and Springer Link. Next, we formulated a search query to retrieve only the relevant studies focusing on deep learning for malware and intrusion detection. We identified keywords, based on the research questions that represent and reflect the RQs to objectively search the relevant publications. It is not effective to search the libraries using individual words; instead, the search keywords are combined by using “AND” and “OR” operators to generate queries that may return only relevant results. The search query formulated was finally fed into the searching mechanism of each library to retrieve the relevant studies from the last six years, 2015 to 2022. We searched these libraries for the previous years as well; however, we found very few, or no relevant studies during these years; thus, we limited our search to the recent years only. Moreover, we intend to focus on the latest state-of-the-art research in our study. We selected the primary studies by analyzing the title and abstract of the publications. We created an EndNote library and downloaded all the selected papers which were then read in full to exclude the papers which were not relevant to the theme of the current study and generated a final set of the relevant studies. This set of publications was then studied to answer the research questions (RQs) and to achieve the objectives of the study.A tracer list for the findings and insights of each RQ is given below to easily guide the readers towards the respective section of the paper for better comprehension. Section 3.2.1 describes different platforms and extracts the insights regarding the severely affected platform so that to find answer to RQ1. Section 3.2.1 extracts different malware analytics and tries to find answer to RQ2. Column 3 of Tables2–6 of Sections 4–8, respectively, extracts insights regarding the DL-based malware detection methods and tries to find answer to RQ3, insights for RQ4, which are the major DL algorithms used so far and have been discussed in Section 9. The key research challenges faced during implementation of DL-based malware detection systems are focused in RQ5 and described in Section 11.3. RQ6 and RQ7 describe the issues of sustainability and evolvability which are also discussed in Section 11.3 and RQ8 extracts insights of the most commonly used datasets for malware detection in Section 11.4.Table 2
Summary of the metadata extracted from the literature on Windows-based malware detection.
Ref.Description: method and features used to train and evaluate modelDL algorithm usedLibrary/framework usedPlatformDataset usedAccuracy/F1 score[26]Malware classification by extracting static features and converting to gray imagesCNNNot statedWindowsKaggle by Microsoft98.86%[27]Malware classification by converting malware binary file to gray image through code mapping, texture partitioning, and texture extractionCNNNot statedWindowsBIG 201599%[28]Malware classification by extracting series of system calls having malicious behaviorNot statedNot statedWindowsSelf-generated95.6%[29]Malware detection and classification by using the op-code and API calls data of malware and benign-wareCNN, BPNNNot statedWindowsSelf-generated95%[30]Multilevel deep learning system for malware detection using different static and dynamic featuresProposed MLDLSNot statedWindowsSelf-generatedNot stated[31]Ransomware detection system based onn-gram op-code with deep learningCNNKeras, TensorFlowWindowsSelf-generated89.5%[32]Malware detection by transforming PE file to op-code sequences and representing the op-code asn-gram vectorDBNNot statedWindowsSelf-generatedAbout 98%[33]Malware detection by visualizing the malware binary file as gray imageCNNMatConvNet in MATLABWindowsSelf-generated—[34]Malware detection using API calls of Windows’ executable filesDAE, RBMNot statedWindowsComodo Cloud Security Center’s datasetAround 98%[35]Malware detection based on API calls sequence and statistical featuresLSTM, RNNTensorFlowWindowsSelf-generated95.7%[36]Identifying executable files as malware or benign using static and dynamic analysis and categorizing the malware to the corresponding familyCNN, LSTMTensorFlow, KerasWindowsMalimg, EMBER, self-generated98.8%[37]Hybrid image-based technique for malware detection by converting malware binaries to gray imagesCNN, LSTMTensorFlow, KerasWindowsBIG 201596–97%[38]Malware detection by extracting API call sequences of malware using dynamic analysis and generating feature imagesCNNNot statedWindowsVirusShare datasetAbout 99%[39]Predicting malicious behavior of executable program based on small amount of behavioral data within the first few seconds of executionRNNKeras, scikit-learnWindowsSelf-generated96%[40]DLMD: malware detection technique based on static features using byte and ASM filesCNNPyTorchWindowsBIG 201597.5%[41]Malware detection extracting control flow graph of the sample by lazy binding and transforming it into an imageCNNNot statedWindowsMALICIA, VirusShare, VXHeaven92%–97.7%[42]Deep learning system with two hidden layers for malware detection using dependency of malware sequence and avoiding back-propagationTELMNot statedWindowsKaggle, VXHeavenAbove 99%[43]Malware classification by transforming malware binary file to grayscale imagesCNN, LSTMTensorFlow, KerasWindowsBIG 201598.2%[44]Zero-day malware detection by generating fake malware and learning to distinguish it from the real malwareDAE, DCGANKerasWindowsKaggleAbout 99%[45]Malware detection by visualizing the malware as grayscale imageCNNTensorFlowWindowsMalimg, Microsoft dataset99.97%[46]Detecting threats in the cloud-assisted Internet of things by extracting API calls data from malwareDBNNot statedWindowsVXHeavenUp to 99.78%[47]Malware classification by visualizing the malware as grayscale imageCNNNot statedWindowsMalimg, BIG 201597.5%[48]Malware variants detection by visualizing malware samples as grayscale imagesCNNCaffe NN frameworkWindowsDataset by Vision Research Lab94.5%[49]Malware detection by converting malware executable to grayscale image and using NSGA-II algorithm to deal with data imbalanceCNNTensorFlowWindowsDataset by Vision Research Lab97.6%[50]Malware detection by visualizing the malware sample as a grayscale imageDeep transfer learningNot statedWindowsNot stated99.25%[51]Malware detection by using static analysis to extract features of the malware samplesLSTMKeras, TensorFlowWindowsSelf-generated dataset named MC-dataset-multiclass90.63%[52]Malware detection by visualizing the malware sample as a grayscale imageCNNTensorFlowWindowsMalimg80.5%[53]Malware detection by extracting features, like file activity, registry activity, service activity, processes, runtime DLLs and network activities, etc., and applying big data analytics techniquesNot statedKerasWindowsSelf-generated97%[54]Malware classification by converting malware binaries to Markov imagesCNNKeras, TensorFlowWindowsMicrosoft dataset, Drebin dataset97.3% for Drebin, 99.3% for Microsoft[55]A comparative study of CNN and ELM-based detection systems using malware represented as grayscale imagesCNNKerasWindowsMalimg96.3% for CNN, 97.7% for ELM[56]Metamorphic malware detection using API calls made on the operating systemLSTMKerasWindowsSelf-generated API sequence datasetUp to 98.5%[57]Malware detection by extracting features of PE files, including import functions feature, general information feature, and bytes entropy featureNot statedNot statedWindowsSelf-generatedAUC up to 0.989[58]Cryptomining malware detection by static and dynamic analysis of the op-code sequences of PE filesCNN, LSTM, ATT-LSTMNot statedWindowsSelf-generated97% on average[59]Malware classification using malware samples represented as grayscale imagesCNNKeras, TensorFlowWindowsMalimg99.72%[60]Malware classification by extracting features including API calls, sequence of assembly language instructions, and malware’s binary contentsCNNTensorFlowWindowsKaggle99.7%[61]Image-based malware classification system using an ensemble of CNNCNNTensorFlow, Keras, scikit-learnWindowsMalimg99.5%[62]Malware detection by black-and-white embedding of malware images rather than grayscale to avoid bit loss in byteCNNKeras, TensorFlowWindowsKISA dataset92.8%[63]Malware classification by generating a low-dimensional vector and using op-codes and API function calls to train modelBi-LSTMNot statedWindowsMicrosoft dataset96.8%[64]Malware detection by extracting the API calls sequence and generating the API pixel vector and finally visualizing the malwareCNNNot statedWindowsSelf-generated94.7%Table 3
Summary of the metadata extracted from the literature on android-based malware detection.
Ref.Description: method and features used to train and evaluate modelDL algorithm usedAutomatic DL algorithm selection (yes/no)Ability of identifying new malware? (yes/no)Features engineering method (static/dynamic/hybrid analysis)DL model needs updating? (yes/no)Sustainability/resilience against evolution? (yes/no)Library/framework usedPlatformDataset usedAccuracy/F1 score[65]Malware detection using neural networks andk-means clusteringNot statedNoNoStatic analysisYesNoNot statedAndroidSelf-generated88.0%[66]Malware detection based on API method calls sequence miningCNNNoNoStatic analysisYesNoTensorFlowAndroidMalgenome, Drebin, MalDozerAbout 99%[67]Malware detection by analyzing the permission wanted by appDeep eigenspace learningNoNoStatic analysisYesNoNot statedAndroidSelf-generatedNot stated[68]Malware detection by extracting and analyzing several featuresMultimodal neural networksNoNoStatic analysisYesNoKeras, TensorFlow, scikit-learnAndroidVirusShare, malgenome94 – 98%[69]Dynamic malware detection system based on CPU, memory, and battery usageLSTM RNN, encoder-decoderNoNoStatic analysisYesNoNot statedAndroidM0DroidAbout 80%[70]Malware detection by associating the features from static analysis with the features from dynamic analysisDBNNoNoHybrid analysisYesNoNot statedAndroidSelf-generated, malgenome96.76%[71]Malware detection using several static and dynamic featuresDBNNoNoHybrid analysisYesNoNot statedAndroidSelf-generated96.5%[72]Malware detection by using the importance of words from the apk file of applicationsCNNNoNoStatic + renaming variables and prioritizingYesNoNot statedAndroidSelf-generated92.67%[73]Malware detection by extracting several features for model trainingCNNNoNoStatic analysis for static featuresYesNoKerasAndroidSelf-generated99.25%[74]Malware detection using seven different features of android applicationsDAE, CNNNoNoStatic analysis for 7 categories of static featuresYesNoKeras, TensorFlow, scikit-learnAndroidSelf-generated99.82%[75]ITMF, (image texture median filter) for analyzing and detecting malware on Derbin datasetDBNNoPotential of dynamic activity of malwareStatic analysisYesNoKeras, TensorFlow, scikit-learnAndroidDrebin95.43%[76]Malware detection using static analysis, dynamic analysis, and system callsDBNNoNoHybrid analysisYesNoNot statedAndroidNot stated99.1%[77]Malware detection by extracting the API calls graph of applications and generating graph embeddingCNN, RNNNoNoPseudodynamic analysisYesNoKeras, TensorFlowAndroidAMD dataset, AndroZoo, Drebin, ISCX98.86%[78]Malware detection by examining all execution paths and detecting malicious and benign pathsLSTM RNNNoNoPseudodynamic analysisYesNoTensorFlow, Keras, scikit-learnAndroidAndroZoo,91.42%[79]Malware detection using features extracted from manifest file and through static analysis and various deep learning methodsCNN, DBN, LSTM, DAENoNoStatic analysisYesNoTensorFlow, Keras, theanoAndroidDrebin, VirusShareUp to 93.6%[80]Malware detection by extracting features through dynamic analysis and generating Markov chainsRNN, CNN, LSTMNoNoDynamic analysisYesNoTensorFlow, Keras, scikit-learnAndroidDrebinAround 81%[81]Malware detection by extracting texture fingerprint features and mapping malicious code to grayscale imageDBNNoNoStatic analysisYesNoTheano, GDBN, TensorFlow, Keras, scikit-learnAndroidDrebin95.9%[82]Malware detection by using various features, including hardware components, permissions, filtered intents, API calls, and network addressesLSTM, CNNNoNoStatic analysisYesNoNot statedAndroidDrebinUp to 98.53%[83]Malware detection using LASSO feature shrinkage and selection technique and deep belief networksDBNNoNoStatic analysisYesNoNot statedAndroidDroidWare85.22%[84]Malware detection by generating API images from the sequence of API calls of applicationsDAENoNoDynamic analysisYesNoNot statedAndroidMalgenome, contagio minidump98%[85]Malware detection by using features like permissions and API calls and generating a feature vectorDBNNoNoStatic analysisYesNoTensorFlowAndroidDrebin, VirusTotal, contagio, self-generated99.04%[86]Malware detection by extracting byte code from the.dex file of the android packageDAE, DBN, RNN, LSTM, BiLSTM,NoNoStatic analysisYesNoKeras, theanoAndroidDrebin, AMD, VirusShareUp to 99.9%[87]Malware detection by extracting features like API calls, events and permissions by dynamic analysisNot statedNoNoDynamic analysisYesNoNot statedAndroidSelf-generated98.5%[88]Malware detection by converting static features like permissions API calls and components into a set of semantic featuresGCNNoNoStatic analysisYesNoNot statedAndroidDrebin, AMD, lab-built, AndroZoo, PRAGuardUp to 99.7%[89]Hybrid deep learning for android malware detection using various static and dynamic feature of the applicationDBNNoNoHybrid analysisYesNoTensorFlow, KerasAndroidSelf-generated96.8%[90]Malware detection by using dataset comprising of intent features and permission features extracted from benign and malicious applicationsNot statedNoNoHybrid analysisYesNoNot statedAndroidOmnidroid91%[91]Malware detection by converting the application binary to gray-scale imageNot statedNoNoStatic analysisYesNoNot statedAndroid, iOSAMD, self-generated96.6% for Android, 95.8% for iOS[92]Malware detection by extracting byte code of the application and generating embeddingLSTMNoNoHybrid analysisYesNoNot statedAndroid, IoTSelf-generated98% for Android, 99% for IoT malware[93]Malware detection by extracting 11 static behavioral features and transforming them to a multidimensional vectorDBNNoNoStatic analysisYesNoTensorFlowAndroidSelf-generated, Drebin, etc.Up to 99.5%[94]Malware detection by extracting API sequence and the methods from the DEX file of the application and generating the hot vector of the API sequenceBi-LSTMNoNoStatic analysisYesNoNot statedAndroidAMD97.2%[95]Malware detection in the IoT devices by reading the DEX file of the application as an unsigned vector and converting it to a fixed size by image resampling techniqueCNN, RNN, GRU, LSTM, Bi-LSTMNoNoStatic analysisYesNoKerasAndroid IoT devicesSelf-generatedUp to 95.8%[96]Malware detection by generating the function call graph from the DEX file of the application and the op-code-level FCG featuresLSTMNoNoStatic analysisYesNoKeras, TensorFlowAndroidSelf-generated97%[97]Malware detection by extracting and vectorizing the manifest features and API calls from the binary file of the appCNN, GRU, LSTMNoNoStatic analysisYesNoKeras, TensorFlowAndroidDrebin, genome, contagio, pwnzen, VirusShare96.8%[98]Malware detection by extracting features like permissions, system events, APIs and data flow from the manifest, DEX and layout xml filesMLPNoNoStatic analysisYesNoTensorFlowAndroidSelf-generated94.9%[99]Malware detection by extracting static features (permissions) from the manifest file and then generating feature vectorCNN, DAENoNoStatic analysisYesNoKeras, TensorFlowAndroidCIC and Mal2017, self-generated98.2%Table 4
Summary of the metadata extracted from the literature on IoT-based malware detection.
Ref.Description: method and features used to train and evaluate modelDL algorithm usedLibrary/framework usedTargeted platformDatasetusedAccuracy/F1 score[100]IoT and IoBT malware detection through op-code analysisCNNNot statedIoT, IoBTSelf-generated99.68%[101]Behavior-based deep learning framework for detecting malware in IoT environmentSAEKerasIoTSelf-generatedAbout 98.6%[102]Detecting pirated software and security threats in internet of things environmentCNNTensorFlow, KerasIoTNot stated96% for piracy detection, 97.46% for malware detection[103]Malware detection in the Internet of things environment by decompiling and extracting op-codes from application samplesLSTMTensorFlow, Keras, scikit-learnIoTSelf-generated98.18%[104]Attack detection in industrial Internet of things environmentDAE, DFFNNNot statedIoTNSL-KDD, UNSW-NB15Up to 98.6%[105]Classification of malicious applications in the Internet of things environment by using graph embeddingCNNNot statedIoTSelf-generated88.5%[106]Malware detection in industrial IoT devices by extracting DEX file and converting to java class file and extracting the bytecode from the class fileCNNTensorFlowIoTLeopard mobile dataset98.7%Table 5
Summary of the metalevel analysis of the literature on malware detection in platforms other than Windows, Android, and IoT or multiple platforms.
Ref.Description: method and features used to train and evaluate modelDL algorithm usedLibrary/framework usedTargeted platformDatasetUsedAccuracy/F1 score[107]Malware dynamic behavior classification and family clustering algorithmCNN, GANNot statedWindows, androidDataCon, GreekPwnNot stated[108]Detecting domain generation algorithms (DGAs) and automatically labelling domain names in real trafficLSTM RNNKeras, scikit-learnNot statedALexaBamb, RetroAbout 98%[109]Deep learning-based intrusion detection system for detecting cyber-attacksNot statedTensorFlow, Keras, scikit-learnInternetKDDCup99, NSL KDD, UNSW-NB15, WSN-DS, CICIDS 2017, Kyoto85 – 99%[110]An intrusion detection system to protect in-vehicle network, the controller area network (CAN) busCNNNot stated—Self-generated99%[111]Source-based distributed denial-of-service defense system in fog and cloud computing systemsLSTMKeras, TensorFlowCloud computingHogzilla98.88%[112]A hybrid deep learning-based system for detecting botnetCNN, RNNKeras, TensorFlow, scikit-learnInternetCTU-13, ISOT99.3% (CUT-13) 99.5% (ISOT)[113]Using robust software modeling tool (RSMT) to monitor and characterize the behavior of web based applicationsSAEKeras, TensorFlow, scikit-learnInternetNot statedAbout 92%.[114]Deep multilayer perceptron and RNN-based deep learning system for detecting cloud-based intrusionRNN, LSTMKeras, TensorFlow, TheanoCloud computingNot stated86.9%[115]Malware detection in PDF filesCNNNot statedMultipleSelf-generatedUp to 98.92%[116]Ransomware detection and classification by extracting event sequences during a program executionLSTM, CNNKeras, TensorFlowNot statedSelf-generated99.6%[117]Malware detection in cloud platforms by extracting several features of each process, like CPU usage, memory usage, and disk usageCNNNot statedCloud IaaSSelf-generatedUp to 93%[118]Using ML and DL techniques to distinguish normal traffic from cryptomining traffic by extracting the data flow featuresFully connected CNNKeras, TensorFlow, scikit-learnInternetSelf-generated mining traffic99.98%[119]Malware detection on various platforms, including Windows, Android, IoT, IoBT, and the Internet by vector embeddingLSTMNot statedWindows, Android, IoT, IoBT, InternetVXHeaven, Drebin, Kaggle94.1% on averageTable 6
Summary of the metalevel analysis of the literature on malware detection during the recent years.
Ref.Description: method and features used to train and evaluate modelDL algorithm usedLibrary/framework usedTargeted platformDatasetusedAccuracy/F1 score[120]Visualizing malware binaries as two-dimensional images and feeding to classifier that uses reweighted class-balanced loss functionDensely connected CNN with ReLUKerasWindowsMalimg, BIG 2015, MaleVis98.46%[121]Two-stage hybrid malware detection by extracting op-code by static analysis and then performing dynamic analysis to classify benign filesBi-LSTM, CNNNot statedIoTKISA 2019Up to 95%[122]Malware detection by representing the application as image, extracting the dex file, and grouping the sequence of bytes into grayscale pixelCNNCUDA, TensorFlowAndroidArgus Cyber Security Lab97%[123]Malware detection by using text classification method, using the text sequence of APPs analysis and exploring informationCNNKerasAndroidVarious datasets96.6%[124]Malware detection using dynamic analysis by generating dynamic analysis logs for an APK and transforming the features into a feature vectorCNN with leaky ReLUNot statedAndroidSelf-generated98%[64]Malware detection by visualizing malware as RGB color images using both static and dynamic as well as hybrid analysisCNN (VGG16)Not statedWindowsDataset by VirusSign94.7%[125]Detection of Java bytecode malware using static analysis of the Java program and extracting interprocedural control flow graph from bytecode fileCNNNot statedPlatforms capable of running Java programsSelf-generated98.4%[126]Analysis of behavior of malicious programs based on API call graphs. The detection is based on analyzed patterns of the API callsCNN (used only for discovering common features)Not statedAndroidApps from playstore and VirusShare93.2%[127]Classification and detection of malware using executable and linkable format (ELF) binary file, making use of static, dynamic, and hybrid analysisBi-GRU-CNNKeras, TensorFlow, scikit-learnIoTCollected from various sources98% (detect) 100% (classify)[128]Malware classification by converting the bytecode of methods of the malware into grayscale feature image and analyzing its feasibility based on reconstruction error of AEAE based on CNNTensorFlowAndroidApps from playstore and VirusShare96.2%[129]Distributed deep learning-based model for malware detection using both static and dynamic analysisCNN-BiLSTMNot statedWindowsApps from various sources97%[130]Using DL and model-checking to detect malware by converting source code to format of the model-checker, using both static and dynamic analysisCNNPyTorchIoTNot stated95%[131]Malware detection using static analysis, emphasizing on features extraction from PE filesNot statedKerasWindowsEMBER97.5%


#### 3.1.1. Criteria for the Survey

As deep learning-based security systems and other intelligent software tools are getting more and more popular in the field of computer science, it is important to conduct research about how deep learning technology is performing better than the traditional approach for threat hunting and traditional machine learning-based systems. In this survey, we are interested in analyzing which deep learning algorithms are used in malware detection systems and how they can perform better when compared with traditional machine learning-based systems. We carefully read the selected literature to find out which types of threats and what platforms the researchers are targeting and how accurately the deep learning-based systems can detect new security threats when trained with a training dataset. Moreover, in the case of Android-based malware detection research, the focus is also given to know whether the proposed methods support the characteristics of sustainability, evolvability, automatically picking the most appropriate malware detection DL algorithm, ability of identifying new malware (s), and diverse feature analysis methods, such as static, dynamic, or hybrid. Apart from the above, the survey also discusses the issue of data quality for deep learning-based methods in the form of publicly available malware datasets for research purpose.


#### 3.1.2. Query Formulation

As we discussed earlier, it is not the correct way to search each library using a single keyword each time as it is a very tedious and lengthy process. We need to formulate a search query containing all the important keywords including names, synonyms, and abbreviations, which are connected by the logical operators “OR” and “AND.” For example, different studies may use any of “Convolutional Neural Networks” and “CNN” in their title, so we need to combine these by the “OR” operator. Similarly, we need to have both “Deep Learning” and “Malware Detection” discussed in the abstract or the contents of the paper, so we need to combine these keywords by the “AND” operator. We included different keywords that may appear in the relevant studies, including Deep Learning, Deep Learning algorithms, and malware detection. The final search query we formulated is as follows.(“Deep Learning” OR “Convolutional neural network” OR “Deep belief network” OR “recurrent neural network” OR “CNN” OR “RNN” OR “DBN” OR “LSTM”) AND (“malware”) AND (“detection” OR “detect” OR “identification” OR “identify” OR “classification”)


#### 3.1.3. Searched Libraries

For searching the relevant studies, we select five popular libraries, as follows.(i)
Google Scholar:https://scholar.google.com/(accessed 02-07-2022)(ii)
Science Direct:https://www.sciencedirect.com/(accessed 02-07-2022)(iii)
IEEE Explore:https://ieeexplore.ieee.org/(accessed 03-07-2022)(iv)
ACM:https://dl.acm.org/(accessed 03-01-2021)(v)
Springer Link:https://link.springer.com/(accessed 03-07-2022)The selected libraries were searched using the formulated search query, filtering the search results to include only the relevant papers published during 2015–2022. The search query returned a total of 935 publications. The detailed search results for each library are shown in Table7.Table 7
Summary of the literature extracted from the five scholarly libraries and the process of filtration.
LibraryLibrary URLReturned papersSLR/book/gray literatureIrrelevant papersTotalGoogle Scholarhttps://www.scholar.google.com/3337470180Science Directhttps://www.sciencedirect.com/30611414646IEEE Explorehttps://ieeeexplore.ieee.org/144128547ACMhttps://dl.acm.org/9216Springerhttps://link.springer.com/143458711Total935247398290


#### 3.1.4. Inclusion/Exclusion Criteria

Not all of the returned results could be relevant to the domain of the study. We need to further refine the results to get a final list of the most relevant publications. The inclusion and exclusion rules we applied are summarized in Table8.Table 8
The inclusion/exclusion criteria for the deep learning methods.
Inclusion criteria1The paper must discuss a deep learning-based system for malware or intrusion detection.2The paper must be published in a scholarly journal or be a preprint.3The paper is published from January 2015 till 2022.Exclusion criteria1Papers focusing on economic, business, or legal impacts of malware detection and intrusion detection systems2Gray literature, such as blogs or reports3Papers written in a language other than English4Review papers5Duplicate papers6Papers not published in any scholarly journal, such as conference preceding7The studies that do not focus on deep learning for malware detectionOur search query returned a total of 935 papers on all five libraries. Firstly, we excluded the papers that did not appear relevant based on the title or abstract, the survey papers, book chapters, and the gray literature. Table7 summarizes the results after applying inclusion/exclusion criteria.We downloaded the references of the 290 papers and created an endnote library. There, we further refined the results by removing duplicates, non-journal papers, papers written in a language other than English, and irrelevant papers based on full text. Table9 summarizes the finally selected papers for critical analysis.Table 9
Summary of the selected literature for review.
Total papersNonjournal papersDuplicatesNon-EnglishFull text exclusionFinal total29010164711107Figure1 presents a summary of the research methodology used for research articles retrieval.Figure 1
Research articles retrieval methodology of the systematic literature review.


### 3.2. Metalevel Critical Analysis of the Literature

Once the relevant literature is retrieved and filtered, metalevel analysis is performed from different perspectives. In the first phase, to let the readers know of the importance of the proposed study, statistical analysis of the research work done so far is performed. In the second and third phase, respectively, Windows and Mobile platforms-wise metalevel analysis of the malware detection research is performed. These analyses are summarized in the subsequent sections.


#### 3.2.1. Statistical Analysis

To answer to RQ2 and know about the peak era of research for malware detection using deep learning methods, the most appropriate place for publishing the work and the most affected platform out of the Windows and Mobile platforms, the literature is statistically summarized and presented in the following sections.(1) Year-Wise Distribution of the Papers. We observed that, in recent years, there are a lot of research articles published discussing malware and intrusion detection systems using deep learning. We have only one paper in the years 2015 and 2016 while the number increases as we go upward. This shows that malware detection using deep learning algorithms is an active area of research in computer science. Figure 2 shows the year-wise distribution of the research papers published.Figure 2
Year-wise distribution of the research work on the subject area malware detection using deep learning.(2) Platform-Wise Distribution of the Papers (RQ1). As malware is not limited to a specific platform, different researchers have focused on different platforms for malware detection. To answer the RQ1 (which platforms are affected the most?), analytics have been calculated from the selected papers and the insights are summarized in Figure 3, stating that android and windows are the most widely used platforms on mobile devices and personal computers, respectively, and hence affected the most. Most of the researchers focused on these two platforms. However, the Internet of things environment and web-based malware received less attention despite the fact that malware developers target these platforms frequently. Web-based malicious programs are especially a big threat to the Internet and data security. Figure 3 shows the platform-wise distribution of the papers. Based on the findings of RQ1, Sections 4 and 5 have been given more attention, and hence in-depth critical analysis is performed.Figure 3
Platform-wise distribution of the research papers on the subject area and malware detection using deep learning.(3) Journal-Wise Distribution of the Papers. We retrieved relevant publications from a large number of journals using the selected libraries. The number of papers from individual journals varies from 1 to 13, including arXiv preprints and journals from other publishers like Hindawi, IEEE, ACM, and more. Table 10 contains the journal-wise distribution data of the papers.Table 10
Journal-wise distribution of the papers on malware detection using deep learning.
S. no.JournalNo. of papersReferences1ArXiv preprints11[33, 40, 50, 52, 59, 66, 69, 94, 105, 107, 117]2Academia1[65]3ACM SIGCOMM computer communication review1[71]4Ad hoc networks, 2020—Elsevier2[95, 106]5Alexandria Engineering Journal1[76]6Cluster Computing1[79]7Computer Communications1[63]8Computers and Security13[26, 29, 39, 41, 42, 54, 60, 61, 87, 88, 122, 125, 127]9Cybersecurity1[28]10Digital Investigation1[43]11Engineering Applications of Artificial Intelligence1[80]12Expert Systems with Applications1[30]13Future Generation Computer Systems4[31, 86, 103, 116]14IEEE Access8[36, 99, 102, 108, 109, 114, 118, 128]15IEEE Transactions on Industrial Informatics1[48]16IEEE Transactions on Emerging Topics in Computational Intelligence1[119]17IEEE Transactions on Network Science and Engineering1[98]18IEEE Transactions on Sustainable Computing1[100]19IEEE Transactions on Information Forensics and Security2[68, 97]20IEICE Transactions on Information and Systems1[62]21IET Information Security1[38]22Information Sciences1[44]23International Journal of Advance Soft Computing1[82]24International Journal of Digital Crimes and Forensics1[45]25International Journal of Emerging Technology and Computer Science1[67]26International Journal of Performability Engineering1[73]27Journal of Ambient Intelligence and Humanized Computing1[74]28Journal of computer Virology and Hacking Techniques2[47, 91]29Journal of Computers1[81]30Journal of Grid Computing1[58]31Journal of Information Security and Applications2[37, 104]32Journal of Internet Services and Applications1[113]33Journal of King Saud University—Computer and Information Sciences1[111]34Journal of Parallel and Distributed Computing3[46, 49, 84]35Journal of Signal Processing Systems, 2021—Springer1[64]36Knowledge and Information Systems1[34]37KSII Transactions on Internet and Information Systems2[75, 85]38Mathematical Problems in Engineering1[101]39Microelectronics Reliability1[72]40Neural Computing and Applications3[32, 92, 112]41Neurocomputing1[78]42PeerJ Computer Science, 20201[56]43Pertanika Journal of Science and Technology1[83]44PloS One, 20201[57]45Procedia Computer Science2[35, 51]46Security and Communication Networks3[27, 89, 115]47Sensors2[96, 130]48Soft Computing2[77, 93]49TechRxiv1[90]50Tsinghua Science and Technology1[70]51Vehicular Communications1[110]52Entropy1[120]53Human-Centric Computing and Information Sciences1[121]54Applied Soft Computing1[123]55Journal of Internet Services and Information Security1[124]56Signal Processing Systems1[64]57International Journal of Information Security1[126]58Cybernetics and Systems1[129]59International Journal of Computer Network & Information Security1[131]


## 3.1. Research Design

Firstly, we do need an assessment to conduct the literature review. We studied a number of papers from various sources, including journals and conferences. We observed that deep learning has been used by a large number of researchers in malware detection, intrusion detection, and other security-related systems. Based on the need assessment, we formulated a number of research questions, mentioned in the introduction section, to show the effectiveness and outcomes of the proposed study.We then selected a number of search libraries to make sure the selected literature is from an authentic source and is reliable. We selected five different libraries, including Google Scholar, Science Direct, IEEE Explore, ACM, and Springer Link. Next, we formulated a search query to retrieve only the relevant studies focusing on deep learning for malware and intrusion detection. We identified keywords, based on the research questions that represent and reflect the RQs to objectively search the relevant publications. It is not effective to search the libraries using individual words; instead, the search keywords are combined by using “AND” and “OR” operators to generate queries that may return only relevant results. The search query formulated was finally fed into the searching mechanism of each library to retrieve the relevant studies from the last six years, 2015 to 2022. We searched these libraries for the previous years as well; however, we found very few, or no relevant studies during these years; thus, we limited our search to the recent years only. Moreover, we intend to focus on the latest state-of-the-art research in our study. We selected the primary studies by analyzing the title and abstract of the publications. We created an EndNote library and downloaded all the selected papers which were then read in full to exclude the papers which were not relevant to the theme of the current study and generated a final set of the relevant studies. This set of publications was then studied to answer the research questions (RQs) and to achieve the objectives of the study.A tracer list for the findings and insights of each RQ is given below to easily guide the readers towards the respective section of the paper for better comprehension. Section 3.2.1 describes different platforms and extracts the insights regarding the severely affected platform so that to find answer to RQ1. Section 3.2.1 extracts different malware analytics and tries to find answer to RQ2. Column 3 of Tables2–6 of Sections 4–8, respectively, extracts insights regarding the DL-based malware detection methods and tries to find answer to RQ3, insights for RQ4, which are the major DL algorithms used so far and have been discussed in Section 9. The key research challenges faced during implementation of DL-based malware detection systems are focused in RQ5 and described in Section 11.3. RQ6 and RQ7 describe the issues of sustainability and evolvability which are also discussed in Section 11.3 and RQ8 extracts insights of the most commonly used datasets for malware detection in Section 11.4.Table 2
Summary of the metadata extracted from the literature on Windows-based malware detection.
Ref.Description: method and features used to train and evaluate modelDL algorithm usedLibrary/framework usedPlatformDataset usedAccuracy/F1 score[26]Malware classification by extracting static features and converting to gray imagesCNNNot statedWindowsKaggle by Microsoft98.86%[27]Malware classification by converting malware binary file to gray image through code mapping, texture partitioning, and texture extractionCNNNot statedWindowsBIG 201599%[28]Malware classification by extracting series of system calls having malicious behaviorNot statedNot statedWindowsSelf-generated95.6%[29]Malware detection and classification by using the op-code and API calls data of malware and benign-wareCNN, BPNNNot statedWindowsSelf-generated95%[30]Multilevel deep learning system for malware detection using different static and dynamic featuresProposed MLDLSNot statedWindowsSelf-generatedNot stated[31]Ransomware detection system based onn-gram op-code with deep learningCNNKeras, TensorFlowWindowsSelf-generated89.5%[32]Malware detection by transforming PE file to op-code sequences and representing the op-code asn-gram vectorDBNNot statedWindowsSelf-generatedAbout 98%[33]Malware detection by visualizing the malware binary file as gray imageCNNMatConvNet in MATLABWindowsSelf-generated—[34]Malware detection using API calls of Windows’ executable filesDAE, RBMNot statedWindowsComodo Cloud Security Center’s datasetAround 98%[35]Malware detection based on API calls sequence and statistical featuresLSTM, RNNTensorFlowWindowsSelf-generated95.7%[36]Identifying executable files as malware or benign using static and dynamic analysis and categorizing the malware to the corresponding familyCNN, LSTMTensorFlow, KerasWindowsMalimg, EMBER, self-generated98.8%[37]Hybrid image-based technique for malware detection by converting malware binaries to gray imagesCNN, LSTMTensorFlow, KerasWindowsBIG 201596–97%[38]Malware detection by extracting API call sequences of malware using dynamic analysis and generating feature imagesCNNNot statedWindowsVirusShare datasetAbout 99%[39]Predicting malicious behavior of executable program based on small amount of behavioral data within the first few seconds of executionRNNKeras, scikit-learnWindowsSelf-generated96%[40]DLMD: malware detection technique based on static features using byte and ASM filesCNNPyTorchWindowsBIG 201597.5%[41]Malware detection extracting control flow graph of the sample by lazy binding and transforming it into an imageCNNNot statedWindowsMALICIA, VirusShare, VXHeaven92%–97.7%[42]Deep learning system with two hidden layers for malware detection using dependency of malware sequence and avoiding back-propagationTELMNot statedWindowsKaggle, VXHeavenAbove 99%[43]Malware classification by transforming malware binary file to grayscale imagesCNN, LSTMTensorFlow, KerasWindowsBIG 201598.2%[44]Zero-day malware detection by generating fake malware and learning to distinguish it from the real malwareDAE, DCGANKerasWindowsKaggleAbout 99%[45]Malware detection by visualizing the malware as grayscale imageCNNTensorFlowWindowsMalimg, Microsoft dataset99.97%[46]Detecting threats in the cloud-assisted Internet of things by extracting API calls data from malwareDBNNot statedWindowsVXHeavenUp to 99.78%[47]Malware classification by visualizing the malware as grayscale imageCNNNot statedWindowsMalimg, BIG 201597.5%[48]Malware variants detection by visualizing malware samples as grayscale imagesCNNCaffe NN frameworkWindowsDataset by Vision Research Lab94.5%[49]Malware detection by converting malware executable to grayscale image and using NSGA-II algorithm to deal with data imbalanceCNNTensorFlowWindowsDataset by Vision Research Lab97.6%[50]Malware detection by visualizing the malware sample as a grayscale imageDeep transfer learningNot statedWindowsNot stated99.25%[51]Malware detection by using static analysis to extract features of the malware samplesLSTMKeras, TensorFlowWindowsSelf-generated dataset named MC-dataset-multiclass90.63%[52]Malware detection by visualizing the malware sample as a grayscale imageCNNTensorFlowWindowsMalimg80.5%[53]Malware detection by extracting features, like file activity, registry activity, service activity, processes, runtime DLLs and network activities, etc., and applying big data analytics techniquesNot statedKerasWindowsSelf-generated97%[54]Malware classification by converting malware binaries to Markov imagesCNNKeras, TensorFlowWindowsMicrosoft dataset, Drebin dataset97.3% for Drebin, 99.3% for Microsoft[55]A comparative study of CNN and ELM-based detection systems using malware represented as grayscale imagesCNNKerasWindowsMalimg96.3% for CNN, 97.7% for ELM[56]Metamorphic malware detection using API calls made on the operating systemLSTMKerasWindowsSelf-generated API sequence datasetUp to 98.5%[57]Malware detection by extracting features of PE files, including import functions feature, general information feature, and bytes entropy featureNot statedNot statedWindowsSelf-generatedAUC up to 0.989[58]Cryptomining malware detection by static and dynamic analysis of the op-code sequences of PE filesCNN, LSTM, ATT-LSTMNot statedWindowsSelf-generated97% on average[59]Malware classification using malware samples represented as grayscale imagesCNNKeras, TensorFlowWindowsMalimg99.72%[60]Malware classification by extracting features including API calls, sequence of assembly language instructions, and malware’s binary contentsCNNTensorFlowWindowsKaggle99.7%[61]Image-based malware classification system using an ensemble of CNNCNNTensorFlow, Keras, scikit-learnWindowsMalimg99.5%[62]Malware detection by black-and-white embedding of malware images rather than grayscale to avoid bit loss in byteCNNKeras, TensorFlowWindowsKISA dataset92.8%[63]Malware classification by generating a low-dimensional vector and using op-codes and API function calls to train modelBi-LSTMNot statedWindowsMicrosoft dataset96.8%[64]Malware detection by extracting the API calls sequence and generating the API pixel vector and finally visualizing the malwareCNNNot statedWindowsSelf-generated94.7%Table 3
Summary of the metadata extracted from the literature on android-based malware detection.
Ref.Description: method and features used to train and evaluate modelDL algorithm usedAutomatic DL algorithm selection (yes/no)Ability of identifying new malware? (yes/no)Features engineering method (static/dynamic/hybrid analysis)DL model needs updating? (yes/no)Sustainability/resilience against evolution? (yes/no)Library/framework usedPlatformDataset usedAccuracy/F1 score[65]Malware detection using neural networks andk-means clusteringNot statedNoNoStatic analysisYesNoNot statedAndroidSelf-generated88.0%[66]Malware detection based on API method calls sequence miningCNNNoNoStatic analysisYesNoTensorFlowAndroidMalgenome, Drebin, MalDozerAbout 99%[67]Malware detection by analyzing the permission wanted by appDeep eigenspace learningNoNoStatic analysisYesNoNot statedAndroidSelf-generatedNot stated[68]Malware detection by extracting and analyzing several featuresMultimodal neural networksNoNoStatic analysisYesNoKeras, TensorFlow, scikit-learnAndroidVirusShare, malgenome94 – 98%[69]Dynamic malware detection system based on CPU, memory, and battery usageLSTM RNN, encoder-decoderNoNoStatic analysisYesNoNot statedAndroidM0DroidAbout 80%[70]Malware detection by associating the features from static analysis with the features from dynamic analysisDBNNoNoHybrid analysisYesNoNot statedAndroidSelf-generated, malgenome96.76%[71]Malware detection using several static and dynamic featuresDBNNoNoHybrid analysisYesNoNot statedAndroidSelf-generated96.5%[72]Malware detection by using the importance of words from the apk file of applicationsCNNNoNoStatic + renaming variables and prioritizingYesNoNot statedAndroidSelf-generated92.67%[73]Malware detection by extracting several features for model trainingCNNNoNoStatic analysis for static featuresYesNoKerasAndroidSelf-generated99.25%[74]Malware detection using seven different features of android applicationsDAE, CNNNoNoStatic analysis for 7 categories of static featuresYesNoKeras, TensorFlow, scikit-learnAndroidSelf-generated99.82%[75]ITMF, (image texture median filter) for analyzing and detecting malware on Derbin datasetDBNNoPotential of dynamic activity of malwareStatic analysisYesNoKeras, TensorFlow, scikit-learnAndroidDrebin95.43%[76]Malware detection using static analysis, dynamic analysis, and system callsDBNNoNoHybrid analysisYesNoNot statedAndroidNot stated99.1%[77]Malware detection by extracting the API calls graph of applications and generating graph embeddingCNN, RNNNoNoPseudodynamic analysisYesNoKeras, TensorFlowAndroidAMD dataset, AndroZoo, Drebin, ISCX98.86%[78]Malware detection by examining all execution paths and detecting malicious and benign pathsLSTM RNNNoNoPseudodynamic analysisYesNoTensorFlow, Keras, scikit-learnAndroidAndroZoo,91.42%[79]Malware detection using features extracted from manifest file and through static analysis and various deep learning methodsCNN, DBN, LSTM, DAENoNoStatic analysisYesNoTensorFlow, Keras, theanoAndroidDrebin, VirusShareUp to 93.6%[80]Malware detection by extracting features through dynamic analysis and generating Markov chainsRNN, CNN, LSTMNoNoDynamic analysisYesNoTensorFlow, Keras, scikit-learnAndroidDrebinAround 81%[81]Malware detection by extracting texture fingerprint features and mapping malicious code to grayscale imageDBNNoNoStatic analysisYesNoTheano, GDBN, TensorFlow, Keras, scikit-learnAndroidDrebin95.9%[82]Malware detection by using various features, including hardware components, permissions, filtered intents, API calls, and network addressesLSTM, CNNNoNoStatic analysisYesNoNot statedAndroidDrebinUp to 98.53%[83]Malware detection using LASSO feature shrinkage and selection technique and deep belief networksDBNNoNoStatic analysisYesNoNot statedAndroidDroidWare85.22%[84]Malware detection by generating API images from the sequence of API calls of applicationsDAENoNoDynamic analysisYesNoNot statedAndroidMalgenome, contagio minidump98%[85]Malware detection by using features like permissions and API calls and generating a feature vectorDBNNoNoStatic analysisYesNoTensorFlowAndroidDrebin, VirusTotal, contagio, self-generated99.04%[86]Malware detection by extracting byte code from the.dex file of the android packageDAE, DBN, RNN, LSTM, BiLSTM,NoNoStatic analysisYesNoKeras, theanoAndroidDrebin, AMD, VirusShareUp to 99.9%[87]Malware detection by extracting features like API calls, events and permissions by dynamic analysisNot statedNoNoDynamic analysisYesNoNot statedAndroidSelf-generated98.5%[88]Malware detection by converting static features like permissions API calls and components into a set of semantic featuresGCNNoNoStatic analysisYesNoNot statedAndroidDrebin, AMD, lab-built, AndroZoo, PRAGuardUp to 99.7%[89]Hybrid deep learning for android malware detection using various static and dynamic feature of the applicationDBNNoNoHybrid analysisYesNoTensorFlow, KerasAndroidSelf-generated96.8%[90]Malware detection by using dataset comprising of intent features and permission features extracted from benign and malicious applicationsNot statedNoNoHybrid analysisYesNoNot statedAndroidOmnidroid91%[91]Malware detection by converting the application binary to gray-scale imageNot statedNoNoStatic analysisYesNoNot statedAndroid, iOSAMD, self-generated96.6% for Android, 95.8% for iOS[92]Malware detection by extracting byte code of the application and generating embeddingLSTMNoNoHybrid analysisYesNoNot statedAndroid, IoTSelf-generated98% for Android, 99% for IoT malware[93]Malware detection by extracting 11 static behavioral features and transforming them to a multidimensional vectorDBNNoNoStatic analysisYesNoTensorFlowAndroidSelf-generated, Drebin, etc.Up to 99.5%[94]Malware detection by extracting API sequence and the methods from the DEX file of the application and generating the hot vector of the API sequenceBi-LSTMNoNoStatic analysisYesNoNot statedAndroidAMD97.2%[95]Malware detection in the IoT devices by reading the DEX file of the application as an unsigned vector and converting it to a fixed size by image resampling techniqueCNN, RNN, GRU, LSTM, Bi-LSTMNoNoStatic analysisYesNoKerasAndroid IoT devicesSelf-generatedUp to 95.8%[96]Malware detection by generating the function call graph from the DEX file of the application and the op-code-level FCG featuresLSTMNoNoStatic analysisYesNoKeras, TensorFlowAndroidSelf-generated97%[97]Malware detection by extracting and vectorizing the manifest features and API calls from the binary file of the appCNN, GRU, LSTMNoNoStatic analysisYesNoKeras, TensorFlowAndroidDrebin, genome, contagio, pwnzen, VirusShare96.8%[98]Malware detection by extracting features like permissions, system events, APIs and data flow from the manifest, DEX and layout xml filesMLPNoNoStatic analysisYesNoTensorFlowAndroidSelf-generated94.9%[99]Malware detection by extracting static features (permissions) from the manifest file and then generating feature vectorCNN, DAENoNoStatic analysisYesNoKeras, TensorFlowAndroidCIC and Mal2017, self-generated98.2%Table 4
Summary of the metadata extracted from the literature on IoT-based malware detection.
Ref.Description: method and features used to train and evaluate modelDL algorithm usedLibrary/framework usedTargeted platformDatasetusedAccuracy/F1 score[100]IoT and IoBT malware detection through op-code analysisCNNNot statedIoT, IoBTSelf-generated99.68%[101]Behavior-based deep learning framework for detecting malware in IoT environmentSAEKerasIoTSelf-generatedAbout 98.6%[102]Detecting pirated software and security threats in internet of things environmentCNNTensorFlow, KerasIoTNot stated96% for piracy detection, 97.46% for malware detection[103]Malware detection in the Internet of things environment by decompiling and extracting op-codes from application samplesLSTMTensorFlow, Keras, scikit-learnIoTSelf-generated98.18%[104]Attack detection in industrial Internet of things environmentDAE, DFFNNNot statedIoTNSL-KDD, UNSW-NB15Up to 98.6%[105]Classification of malicious applications in the Internet of things environment by using graph embeddingCNNNot statedIoTSelf-generated88.5%[106]Malware detection in industrial IoT devices by extracting DEX file and converting to java class file and extracting the bytecode from the class fileCNNTensorFlowIoTLeopard mobile dataset98.7%Table 5
Summary of the metalevel analysis of the literature on malware detection in platforms other than Windows, Android, and IoT or multiple platforms.
Ref.Description: method and features used to train and evaluate modelDL algorithm usedLibrary/framework usedTargeted platformDatasetUsedAccuracy/F1 score[107]Malware dynamic behavior classification and family clustering algorithmCNN, GANNot statedWindows, androidDataCon, GreekPwnNot stated[108]Detecting domain generation algorithms (DGAs) and automatically labelling domain names in real trafficLSTM RNNKeras, scikit-learnNot statedALexaBamb, RetroAbout 98%[109]Deep learning-based intrusion detection system for detecting cyber-attacksNot statedTensorFlow, Keras, scikit-learnInternetKDDCup99, NSL KDD, UNSW-NB15, WSN-DS, CICIDS 2017, Kyoto85 – 99%[110]An intrusion detection system to protect in-vehicle network, the controller area network (CAN) busCNNNot stated—Self-generated99%[111]Source-based distributed denial-of-service defense system in fog and cloud computing systemsLSTMKeras, TensorFlowCloud computingHogzilla98.88%[112]A hybrid deep learning-based system for detecting botnetCNN, RNNKeras, TensorFlow, scikit-learnInternetCTU-13, ISOT99.3% (CUT-13) 99.5% (ISOT)[113]Using robust software modeling tool (RSMT) to monitor and characterize the behavior of web based applicationsSAEKeras, TensorFlow, scikit-learnInternetNot statedAbout 92%.[114]Deep multilayer perceptron and RNN-based deep learning system for detecting cloud-based intrusionRNN, LSTMKeras, TensorFlow, TheanoCloud computingNot stated86.9%[115]Malware detection in PDF filesCNNNot statedMultipleSelf-generatedUp to 98.92%[116]Ransomware detection and classification by extracting event sequences during a program executionLSTM, CNNKeras, TensorFlowNot statedSelf-generated99.6%[117]Malware detection in cloud platforms by extracting several features of each process, like CPU usage, memory usage, and disk usageCNNNot statedCloud IaaSSelf-generatedUp to 93%[118]Using ML and DL techniques to distinguish normal traffic from cryptomining traffic by extracting the data flow featuresFully connected CNNKeras, TensorFlow, scikit-learnInternetSelf-generated mining traffic99.98%[119]Malware detection on various platforms, including Windows, Android, IoT, IoBT, and the Internet by vector embeddingLSTMNot statedWindows, Android, IoT, IoBT, InternetVXHeaven, Drebin, Kaggle94.1% on averageTable 6
Summary of the metalevel analysis of the literature on malware detection during the recent years.
Ref.Description: method and features used to train and evaluate modelDL algorithm usedLibrary/framework usedTargeted platformDatasetusedAccuracy/F1 score[120]Visualizing malware binaries as two-dimensional images and feeding to classifier that uses reweighted class-balanced loss functionDensely connected CNN with ReLUKerasWindowsMalimg, BIG 2015, MaleVis98.46%[121]Two-stage hybrid malware detection by extracting op-code by static analysis and then performing dynamic analysis to classify benign filesBi-LSTM, CNNNot statedIoTKISA 2019Up to 95%[122]Malware detection by representing the application as image, extracting the dex file, and grouping the sequence of bytes into grayscale pixelCNNCUDA, TensorFlowAndroidArgus Cyber Security Lab97%[123]Malware detection by using text classification method, using the text sequence of APPs analysis and exploring informationCNNKerasAndroidVarious datasets96.6%[124]Malware detection using dynamic analysis by generating dynamic analysis logs for an APK and transforming the features into a feature vectorCNN with leaky ReLUNot statedAndroidSelf-generated98%[64]Malware detection by visualizing malware as RGB color images using both static and dynamic as well as hybrid analysisCNN (VGG16)Not statedWindowsDataset by VirusSign94.7%[125]Detection of Java bytecode malware using static analysis of the Java program and extracting interprocedural control flow graph from bytecode fileCNNNot statedPlatforms capable of running Java programsSelf-generated98.4%[126]Analysis of behavior of malicious programs based on API call graphs. The detection is based on analyzed patterns of the API callsCNN (used only for discovering common features)Not statedAndroidApps from playstore and VirusShare93.2%[127]Classification and detection of malware using executable and linkable format (ELF) binary file, making use of static, dynamic, and hybrid analysisBi-GRU-CNNKeras, TensorFlow, scikit-learnIoTCollected from various sources98% (detect) 100% (classify)[128]Malware classification by converting the bytecode of methods of the malware into grayscale feature image and analyzing its feasibility based on reconstruction error of AEAE based on CNNTensorFlowAndroidApps from playstore and VirusShare96.2%[129]Distributed deep learning-based model for malware detection using both static and dynamic analysisCNN-BiLSTMNot statedWindowsApps from various sources97%[130]Using DL and model-checking to detect malware by converting source code to format of the model-checker, using both static and dynamic analysisCNNPyTorchIoTNot stated95%[131]Malware detection using static analysis, emphasizing on features extraction from PE filesNot statedKerasWindowsEMBER97.5%


### 3.1.1. Criteria for the Survey

As deep learning-based security systems and other intelligent software tools are getting more and more popular in the field of computer science, it is important to conduct research about how deep learning technology is performing better than the traditional approach for threat hunting and traditional machine learning-based systems. In this survey, we are interested in analyzing which deep learning algorithms are used in malware detection systems and how they can perform better when compared with traditional machine learning-based systems. We carefully read the selected literature to find out which types of threats and what platforms the researchers are targeting and how accurately the deep learning-based systems can detect new security threats when trained with a training dataset. Moreover, in the case of Android-based malware detection research, the focus is also given to know whether the proposed methods support the characteristics of sustainability, evolvability, automatically picking the most appropriate malware detection DL algorithm, ability of identifying new malware (s), and diverse feature analysis methods, such as static, dynamic, or hybrid. Apart from the above, the survey also discusses the issue of data quality for deep learning-based methods in the form of publicly available malware datasets for research purpose.


### 3.1.2. Query Formulation

As we discussed earlier, it is not the correct way to search each library using a single keyword each time as it is a very tedious and lengthy process. We need to formulate a search query containing all the important keywords including names, synonyms, and abbreviations, which are connected by the logical operators “OR” and “AND.” For example, different studies may use any of “Convolutional Neural Networks” and “CNN” in their title, so we need to combine these by the “OR” operator. Similarly, we need to have both “Deep Learning” and “Malware Detection” discussed in the abstract or the contents of the paper, so we need to combine these keywords by the “AND” operator. We included different keywords that may appear in the relevant studies, including Deep Learning, Deep Learning algorithms, and malware detection. The final search query we formulated is as follows.(“Deep Learning” OR “Convolutional neural network” OR “Deep belief network” OR “recurrent neural network” OR “CNN” OR “RNN” OR “DBN” OR “LSTM”) AND (“malware”) AND (“detection” OR “detect” OR “identification” OR “identify” OR “classification”)


### 3.1.3. Searched Libraries

For searching the relevant studies, we select five popular libraries, as follows.(i)
Google Scholar:https://scholar.google.com/(accessed 02-07-2022)(ii)
Science Direct:https://www.sciencedirect.com/(accessed 02-07-2022)(iii)
IEEE Explore:https://ieeexplore.ieee.org/(accessed 03-07-2022)(iv)
ACM:https://dl.acm.org/(accessed 03-01-2021)(v)
Springer Link:https://link.springer.com/(accessed 03-07-2022)The selected libraries were searched using the formulated search query, filtering the search results to include only the relevant papers published during 2015–2022. The search query returned a total of 935 publications. The detailed search results for each library are shown in Table7.Table 7
Summary of the literature extracted from the five scholarly libraries and the process of filtration.
LibraryLibrary URLReturned papersSLR/book/gray literatureIrrelevant papersTotalGoogle Scholarhttps://www.scholar.google.com/3337470180Science Directhttps://www.sciencedirect.com/30611414646IEEE Explorehttps://ieeeexplore.ieee.org/144128547ACMhttps://dl.acm.org/9216Springerhttps://link.springer.com/143458711Total935247398290


### 3.1.4. Inclusion/Exclusion Criteria

Not all of the returned results could be relevant to the domain of the study. We need to further refine the results to get a final list of the most relevant publications. The inclusion and exclusion rules we applied are summarized in Table8.Table 8
The inclusion/exclusion criteria for the deep learning methods.
Inclusion criteria1The paper must discuss a deep learning-based system for malware or intrusion detection.2The paper must be published in a scholarly journal or be a preprint.3The paper is published from January 2015 till 2022.Exclusion criteria1Papers focusing on economic, business, or legal impacts of malware detection and intrusion detection systems2Gray literature, such as blogs or reports3Papers written in a language other than English4Review papers5Duplicate papers6Papers not published in any scholarly journal, such as conference preceding7The studies that do not focus on deep learning for malware detectionOur search query returned a total of 935 papers on all five libraries. Firstly, we excluded the papers that did not appear relevant based on the title or abstract, the survey papers, book chapters, and the gray literature. Table7 summarizes the results after applying inclusion/exclusion criteria.We downloaded the references of the 290 papers and created an endnote library. There, we further refined the results by removing duplicates, non-journal papers, papers written in a language other than English, and irrelevant papers based on full text. Table9 summarizes the finally selected papers for critical analysis.Table 9
Summary of the selected literature for review.
Total papersNonjournal papersDuplicatesNon-EnglishFull text exclusionFinal total29010164711107Figure1 presents a summary of the research methodology used for research articles retrieval.Figure 1
Research articles retrieval methodology of the systematic literature review.


## 3.1.1. Criteria for the Survey

As deep learning-based security systems and other intelligent software tools are getting more and more popular in the field of computer science, it is important to conduct research about how deep learning technology is performing better than the traditional approach for threat hunting and traditional machine learning-based systems. In this survey, we are interested in analyzing which deep learning algorithms are used in malware detection systems and how they can perform better when compared with traditional machine learning-based systems. We carefully read the selected literature to find out which types of threats and what platforms the researchers are targeting and how accurately the deep learning-based systems can detect new security threats when trained with a training dataset. Moreover, in the case of Android-based malware detection research, the focus is also given to know whether the proposed methods support the characteristics of sustainability, evolvability, automatically picking the most appropriate malware detection DL algorithm, ability of identifying new malware (s), and diverse feature analysis methods, such as static, dynamic, or hybrid. Apart from the above, the survey also discusses the issue of data quality for deep learning-based methods in the form of publicly available malware datasets for research purpose.


## 3.1.2. Query Formulation

As we discussed earlier, it is not the correct way to search each library using a single keyword each time as it is a very tedious and lengthy process. We need to formulate a search query containing all the important keywords including names, synonyms, and abbreviations, which are connected by the logical operators “OR” and “AND.” For example, different studies may use any of “Convolutional Neural Networks” and “CNN” in their title, so we need to combine these by the “OR” operator. Similarly, we need to have both “Deep Learning” and “Malware Detection” discussed in the abstract or the contents of the paper, so we need to combine these keywords by the “AND” operator. We included different keywords that may appear in the relevant studies, including Deep Learning, Deep Learning algorithms, and malware detection. The final search query we formulated is as follows.(“Deep Learning” OR “Convolutional neural network” OR “Deep belief network” OR “recurrent neural network” OR “CNN” OR “RNN” OR “DBN” OR “LSTM”) AND (“malware”) AND (“detection” OR “detect” OR “identification” OR “identify” OR “classification”)


## 3.1.3. Searched Libraries

For searching the relevant studies, we select five popular libraries, as follows.(i)
Google Scholar:https://scholar.google.com/(accessed 02-07-2022)(ii)
Science Direct:https://www.sciencedirect.com/(accessed 02-07-2022)(iii)
IEEE Explore:https://ieeexplore.ieee.org/(accessed 03-07-2022)(iv)
ACM:https://dl.acm.org/(accessed 03-01-2021)(v)
Springer Link:https://link.springer.com/(accessed 03-07-2022)The selected libraries were searched using the formulated search query, filtering the search results to include only the relevant papers published during 2015–2022. The search query returned a total of 935 publications. The detailed search results for each library are shown in Table7.Table 7
Summary of the literature extracted from the five scholarly libraries and the process of filtration.
LibraryLibrary URLReturned papersSLR/book/gray literatureIrrelevant papersTotalGoogle Scholarhttps://www.scholar.google.com/3337470180Science Directhttps://www.sciencedirect.com/30611414646IEEE Explorehttps://ieeeexplore.ieee.org/144128547ACMhttps://dl.acm.org/9216Springerhttps://link.springer.com/143458711Total935247398290


## 3.1.4. Inclusion/Exclusion Criteria

Not all of the returned results could be relevant to the domain of the study. We need to further refine the results to get a final list of the most relevant publications. The inclusion and exclusion rules we applied are summarized in Table8.Table 8
The inclusion/exclusion criteria for the deep learning methods.
Inclusion criteria1The paper must discuss a deep learning-based system for malware or intrusion detection.2The paper must be published in a scholarly journal or be a preprint.3The paper is published from January 2015 till 2022.Exclusion criteria1Papers focusing on economic, business, or legal impacts of malware detection and intrusion detection systems2Gray literature, such as blogs or reports3Papers written in a language other than English4Review papers5Duplicate papers6Papers not published in any scholarly journal, such as conference preceding7The studies that do not focus on deep learning for malware detectionOur search query returned a total of 935 papers on all five libraries. Firstly, we excluded the papers that did not appear relevant based on the title or abstract, the survey papers, book chapters, and the gray literature. Table7 summarizes the results after applying inclusion/exclusion criteria.We downloaded the references of the 290 papers and created an endnote library. There, we further refined the results by removing duplicates, non-journal papers, papers written in a language other than English, and irrelevant papers based on full text. Table9 summarizes the finally selected papers for critical analysis.Table 9
Summary of the selected literature for review.
Total papersNonjournal papersDuplicatesNon-EnglishFull text exclusionFinal total29010164711107Figure1 presents a summary of the research methodology used for research articles retrieval.Figure 1
Research articles retrieval methodology of the systematic literature review.


## 3.2. Metalevel Critical Analysis of the Literature

Once the relevant literature is retrieved and filtered, metalevel analysis is performed from different perspectives. In the first phase, to let the readers know of the importance of the proposed study, statistical analysis of the research work done so far is performed. In the second and third phase, respectively, Windows and Mobile platforms-wise metalevel analysis of the malware detection research is performed. These analyses are summarized in the subsequent sections.


### 3.2.1. Statistical Analysis

To answer to RQ2 and know about the peak era of research for malware detection using deep learning methods, the most appropriate place for publishing the work and the most affected platform out of the Windows and Mobile platforms, the literature is statistically summarized and presented in the following sections.(1) Year-Wise Distribution of the Papers. We observed that, in recent years, there are a lot of research articles published discussing malware and intrusion detection systems using deep learning. We have only one paper in the years 2015 and 2016 while the number increases as we go upward. This shows that malware detection using deep learning algorithms is an active area of research in computer science. Figure 2 shows the year-wise distribution of the research papers published.Figure 2
Year-wise distribution of the research work on the subject area malware detection using deep learning.(2) Platform-Wise Distribution of the Papers (RQ1). As malware is not limited to a specific platform, different researchers have focused on different platforms for malware detection. To answer the RQ1 (which platforms are affected the most?), analytics have been calculated from the selected papers and the insights are summarized in Figure 3, stating that android and windows are the most widely used platforms on mobile devices and personal computers, respectively, and hence affected the most. Most of the researchers focused on these two platforms. However, the Internet of things environment and web-based malware received less attention despite the fact that malware developers target these platforms frequently. Web-based malicious programs are especially a big threat to the Internet and data security. Figure 3 shows the platform-wise distribution of the papers. Based on the findings of RQ1, Sections 4 and 5 have been given more attention, and hence in-depth critical analysis is performed.Figure 3
Platform-wise distribution of the research papers on the subject area and malware detection using deep learning.(3) Journal-Wise Distribution of the Papers. We retrieved relevant publications from a large number of journals using the selected libraries. The number of papers from individual journals varies from 1 to 13, including arXiv preprints and journals from other publishers like Hindawi, IEEE, ACM, and more. Table 10 contains the journal-wise distribution data of the papers.Table 10
Journal-wise distribution of the papers on malware detection using deep learning.
S. no.JournalNo. of papersReferences1ArXiv preprints11[33, 40, 50, 52, 59, 66, 69, 94, 105, 107, 117]2Academia1[65]3ACM SIGCOMM computer communication review1[71]4Ad hoc networks, 2020—Elsevier2[95, 106]5Alexandria Engineering Journal1[76]6Cluster Computing1[79]7Computer Communications1[63]8Computers and Security13[26, 29, 39, 41, 42, 54, 60, 61, 87, 88, 122, 125, 127]9Cybersecurity1[28]10Digital Investigation1[43]11Engineering Applications of Artificial Intelligence1[80]12Expert Systems with Applications1[30]13Future Generation Computer Systems4[31, 86, 103, 116]14IEEE Access8[36, 99, 102, 108, 109, 114, 118, 128]15IEEE Transactions on Industrial Informatics1[48]16IEEE Transactions on Emerging Topics in Computational Intelligence1[119]17IEEE Transactions on Network Science and Engineering1[98]18IEEE Transactions on Sustainable Computing1[100]19IEEE Transactions on Information Forensics and Security2[68, 97]20IEICE Transactions on Information and Systems1[62]21IET Information Security1[38]22Information Sciences1[44]23International Journal of Advance Soft Computing1[82]24International Journal of Digital Crimes and Forensics1[45]25International Journal of Emerging Technology and Computer Science1[67]26International Journal of Performability Engineering1[73]27Journal of Ambient Intelligence and Humanized Computing1[74]28Journal of computer Virology and Hacking Techniques2[47, 91]29Journal of Computers1[81]30Journal of Grid Computing1[58]31Journal of Information Security and Applications2[37, 104]32Journal of Internet Services and Applications1[113]33Journal of King Saud University—Computer and Information Sciences1[111]34Journal of Parallel and Distributed Computing3[46, 49, 84]35Journal of Signal Processing Systems, 2021—Springer1[64]36Knowledge and Information Systems1[34]37KSII Transactions on Internet and Information Systems2[75, 85]38Mathematical Problems in Engineering1[101]39Microelectronics Reliability1[72]40Neural Computing and Applications3[32, 92, 112]41Neurocomputing1[78]42PeerJ Computer Science, 20201[56]43Pertanika Journal of Science and Technology1[83]44PloS One, 20201[57]45Procedia Computer Science2[35, 51]46Security and Communication Networks3[27, 89, 115]47Sensors2[96, 130]48Soft Computing2[77, 93]49TechRxiv1[90]50Tsinghua Science and Technology1[70]51Vehicular Communications1[110]52Entropy1[120]53Human-Centric Computing and Information Sciences1[121]54Applied Soft Computing1[123]55Journal of Internet Services and Information Security1[124]56Signal Processing Systems1[64]57International Journal of Information Security1[126]58Cybernetics and Systems1[129]59International Journal of Computer Network & Information Security1[131]


## 3.2.1. Statistical Analysis

To answer to RQ2 and know about the peak era of research for malware detection using deep learning methods, the most appropriate place for publishing the work and the most affected platform out of the Windows and Mobile platforms, the literature is statistically summarized and presented in the following sections.(1) Year-Wise Distribution of the Papers. We observed that, in recent years, there are a lot of research articles published discussing malware and intrusion detection systems using deep learning. We have only one paper in the years 2015 and 2016 while the number increases as we go upward. This shows that malware detection using deep learning algorithms is an active area of research in computer science. Figure 2 shows the year-wise distribution of the research papers published.Figure 2
Year-wise distribution of the research work on the subject area malware detection using deep learning.(2) Platform-Wise Distribution of the Papers (RQ1). As malware is not limited to a specific platform, different researchers have focused on different platforms for malware detection. To answer the RQ1 (which platforms are affected the most?), analytics have been calculated from the selected papers and the insights are summarized in Figure 3, stating that android and windows are the most widely used platforms on mobile devices and personal computers, respectively, and hence affected the most. Most of the researchers focused on these two platforms. However, the Internet of things environment and web-based malware received less attention despite the fact that malware developers target these platforms frequently. Web-based malicious programs are especially a big threat to the Internet and data security. Figure 3 shows the platform-wise distribution of the papers. Based on the findings of RQ1, Sections 4 and 5 have been given more attention, and hence in-depth critical analysis is performed.Figure 3
Platform-wise distribution of the research papers on the subject area and malware detection using deep learning.(3) Journal-Wise Distribution of the Papers. We retrieved relevant publications from a large number of journals using the selected libraries. The number of papers from individual journals varies from 1 to 13, including arXiv preprints and journals from other publishers like Hindawi, IEEE, ACM, and more. Table 10 contains the journal-wise distribution data of the papers.Table 10
Journal-wise distribution of the papers on malware detection using deep learning.
S. no.JournalNo. of papersReferences1ArXiv preprints11[33, 40, 50, 52, 59, 66, 69, 94, 105, 107, 117]2Academia1[65]3ACM SIGCOMM computer communication review1[71]4Ad hoc networks, 2020—Elsevier2[95, 106]5Alexandria Engineering Journal1[76]6Cluster Computing1[79]7Computer Communications1[63]8Computers and Security13[26, 29, 39, 41, 42, 54, 60, 61, 87, 88, 122, 125, 127]9Cybersecurity1[28]10Digital Investigation1[43]11Engineering Applications of Artificial Intelligence1[80]12Expert Systems with Applications1[30]13Future Generation Computer Systems4[31, 86, 103, 116]14IEEE Access8[36, 99, 102, 108, 109, 114, 118, 128]15IEEE Transactions on Industrial Informatics1[48]16IEEE Transactions on Emerging Topics in Computational Intelligence1[119]17IEEE Transactions on Network Science and Engineering1[98]18IEEE Transactions on Sustainable Computing1[100]19IEEE Transactions on Information Forensics and Security2[68, 97]20IEICE Transactions on Information and Systems1[62]21IET Information Security1[38]22Information Sciences1[44]23International Journal of Advance Soft Computing1[82]24International Journal of Digital Crimes and Forensics1[45]25International Journal of Emerging Technology and Computer Science1[67]26International Journal of Performability Engineering1[73]27Journal of Ambient Intelligence and Humanized Computing1[74]28Journal of computer Virology and Hacking Techniques2[47, 91]29Journal of Computers1[81]30Journal of Grid Computing1[58]31Journal of Information Security and Applications2[37, 104]32Journal of Internet Services and Applications1[113]33Journal of King Saud University—Computer and Information Sciences1[111]34Journal of Parallel and Distributed Computing3[46, 49, 84]35Journal of Signal Processing Systems, 2021—Springer1[64]36Knowledge and Information Systems1[34]37KSII Transactions on Internet and Information Systems2[75, 85]38Mathematical Problems in Engineering1[101]39Microelectronics Reliability1[72]40Neural Computing and Applications3[32, 92, 112]41Neurocomputing1[78]42PeerJ Computer Science, 20201[56]43Pertanika Journal of Science and Technology1[83]44PloS One, 20201[57]45Procedia Computer Science2[35, 51]46Security and Communication Networks3[27, 89, 115]47Sensors2[96, 130]48Soft Computing2[77, 93]49TechRxiv1[90]50Tsinghua Science and Technology1[70]51Vehicular Communications1[110]52Entropy1[120]53Human-Centric Computing and Information Sciences1[121]54Applied Soft Computing1[123]55Journal of Internet Services and Information Security1[124]56Signal Processing Systems1[64]57International Journal of Information Security1[126]58Cybernetics and Systems1[129]59International Journal of Computer Network & Information Security1[131]


## 4. Windows Malware Detection

In the Windows platform, researchers have extensively worked on the subject matter to protect personal computers (PC) against cyber-attacks. In this section, we analyze the research work that focuses on Windows malware detection and present a summary in Table2.Ni et al. [26] proposed an algorithm “Malware Classification using SimHash and CNN” (MCSC) based on convolutional neural networks. They disassemble the code of the malware and convert it to gray images to identify its family. They apply locality-sensitive hashing (LSH) to convert similar malware code into similar hash values. These hash values are converted to gray images to train neural networks. They claim about 98% accuracy.Zhao et al. [27] proposed MalDeep, a deep learning-based malware detection system that uses the binary file of the malware. They convert the binary file to gray image and use convolutional neural networks to classify the malware. The strength of their system has a high accuracy of over 99%.Zhang et al. [28] proposed a deep learning system that uses sensitive system calls for malware detection. The application is monitored in Cuckoo sandbox to retrieve system calls data and train the neural networks. Their system achieves an accuracy of over 95%.Zhang et al. [29] proposed a convolutional neural network-based malware detection system that includes unpacking the application to retrieve its op-codes and API calls, generating structured data to represent each binary and obtaining PCA-initialized op-code bi-gram matrix and PCA-initialized API frequency vector which are then fed to CNN and BPNN to train a feature embedding model. Their proposed system achieved an accuracy of 95%.Zhong and Gu [30] proposed a multilevel deep learning system that selects important features from the dynamic and static features set, partitioned the set into many one-level clusters using K-means algorithm, generated cluster subtrees and combined decision values of deep learning models in the tree for classification of the application as malware or benign.Zhang et al. [31] proposed a ransomware detection system that transforms the op-code data and ransomware family label to numeric tensors to be used as input to the neural network. They use self-attention powered convolutional neural networks (SA-CNN) in their proposed method. The weakness of their system is the comparatively lower accuracy of about 90%.Yuxin and Siyi [32] developed a deep belief network-based system that extracts the op-code of malware and used the neural network to detect it. Their system consists of a PE parser that transforms the PE file to op-code sequences, a feature extractor that selects n-grams that have strong classification power and to represent a PE file as n-grams vector, and a malware detection module. Their proposed model achieved about 98% accuracy.Yue [33] proposes a weighted softmax loss (combination of softmax regression and entropy loss) for deep convolutional networks on malware image classification. It is claimed that this would resolve the issues that are caused by the imbalance of malware families.Ye et al. [34] proposed a malware detection system that performs directly on Windows PE file. Their proposed system consists of a feature extractor that decompresses the file and parses PE code to extract the API calls from the file. Then they use unsupervised heterogeneous, autoencoder, and RBM-based deep learning model for malware detection.Xiaofeng et al. [35] combined machine learning and deep learning and proposed an LSTM RNN-based malware detection system that uses API calls sequence and statistical features of malware. They run the malware in a sandbox to get the API calls and use random forest model to classify the system call sequence, which is then processed to get a feature vector that is used as input to the deep learning model for classification.Vinayakumar et al. [36] proposed a distributed system ScalMalNet, which collects malware samples from different sources and processes the malware samples in real time or on demand basis in a distributed manner. They proposed an image processing framework for malware detection and classification using static and dynamic analysis. They applied various shallow learning and deep learning techniques for malware detection and experimentally shown that deep learning-based malware detection systems work much better than traditional ML-based systems.Venkatraman et al. [37] investigated the use of image-based techniques for detecting suspicious behavior and proposed their own image-based malware detection technique by transforming the binary code of malware samples to grayscale images. They use both CNN and LSTM and develop a self-learning system that is capable of detecting the known malware as well as unknown malware.Tang and Qian [38] proposed a CNN-based malware classification system that extracts API calls sequence of the application using dynamic analysis and generating feature image. They run the malware sample in a sandbox and extract the API call sequences and generate feature images using color mapping rules, category of the API and the number of times the category occurs in unit time. These images are used to train the convolutional neural network for detecting unknown malware. The strength of their system is the claimed accuracy of over 99% in most cases.Rhode et al. [39] proposed an early detection system based on a recurrent neural network that works within the first five seconds of execution of a program and detects malicious behavior based on behavioral data. They generate machine activity data metrics based on the initial dynamic data and use them as feature input to the model. The features they used were CPU usage, packets sent and received, bytes sent and received, swap use, memory usage, number of current processes, and the maximum process ID assigned. They compared their RNN model with traditional machine learning algorithms and showed that deep learning performed much better, achieving an accuracy of 96%.Rafique et al. [40] proposed a malware detection technique that uses the byte and ASM files to extract static features for classification. They use a convolutional neural network to extract features from the byte files, while for extracting features from ASM files, a wrapper-based technique is used. Then, they use feature space to train multilayer perceptron, which classifies the different malware categories of the BIG 2015 dataset.Nguyen et al. [41] proposed a CNN-based deep learning system for malware detection that uses a modified form of control flow graph called lazy-binding CFG. They generate CFG from the binary code of the malware by using lazy-binding instead of early-binding for more precise results and convert the CFG to pixel image by transforming the CFG to adjacency matrix. By this way, variants of a specific malware are represented by closely similar objects, which are then inputted to the deep learning model for malware detection.Namavar Jahromi et al. [42] proposed two-hidden-layer-based extreme learning machine (ELM) for malware detection. The system uses dependencies between malware features, like op-codes and API, calls to train the deep learning model. The extreme learning model has a different connection of input to the first hidden layer and is partially connected. They compared their proposed method with various deep learning methods and showed that their method achieved a better accuracy of above 99% in most cases.Le et al. [43] proposed a malware classification system that is based on transforming the malware binaries to grayscale images. They use a convolutional neural network based deep learning model to classify malware, training their model using Microsoft Kaggle dataset. They developed three different models, one with CNN, another one with CNN and LSTM, and the third one with CNN and biLSTM achieving the highest accuracy of 98.2% with the third model.Kim et al. [44] used transferred deep convolutional generative adversarial network (tDCGAN) to detect zero-day (a type of malware) by creating fake malware and feeding it to the modal to learn to distinguish a real malware.Kalash et al. [45] proposed a CNN-based data-independent system for malware detection that uses the grayscale representation of the malware sample. The system reads the malware binary file in a vector of 8-bit integers and converts the binary value to the decimal equivalent and a new decimal vector representation is generated. Then, they represent this decimal vector as a two-dimensional matrix and transforms it to a grayscale image. The strength of their system is the high accuracy rate of 99.97%.Huda et al. [46] proposed a deep belief networks-based system for detecting threats in the cloud-assisted Internet of things environment. To collect data, they execute the malware in a virtual sandbox, observe the change of states and collect any operations performed by it, and generate a report that includes the API calls and their parameters, which are used to prepare a frequency list of the APIs. Next, they train the deep belief networks-based model for malware detection.Gibert et al. [47] proposed a convolutional neural networks-based system for malware classification that visualizes the malware as a grayscale image to extract features. They interpret each byte of the malware sample as a pixel and visualize the resulting two-dimensional array as a grayscale image. Then, they use a convolutional neural network-based system for extracting features from the images. Next, they use CNN with a softmax layer to classify the malware samples.Cui et al. [48] proposed a CNN-based malware variants detection system. They first split the malware binary bit string into 8-bit substrings and consider each of the substrings as a pixel to visualize the image as a grayscale image. Then, they use a convolutional neural network based deep learning system that consists of an input layer, a convolutional layer, a subsampling layer, and several fully connected layers to classify malware.Cui et al. [49] proposed a CNN-based approach for malware detection that uses grayscale images generated from the malware executable. They also split the malware binary bit string into 8-bit substrings each of which is considered as a pixel. This way they visualize the malware as a grayscale image. Then, they use a CNN-based deep learning framework that consists of two convolution layers, one pooling layer, and two dense layers to classify malware samples.Chen [50] proposed a deep transfer learning based method for malware detection. The malware binary was mapped into an integer in the range 0 to 255 to generate pixels and visualize the malware. Next, deep transfer learning-based model was used to classify malware samples. This approach was compared with several shallow learning approaches, and it was shown that deep learning achieved much better results.Andrade et al. [51] developed an LSTM-based system for detecting five different families of malware, including rootkit, virus, Trojan, worm, and backdoor. They rely only on static analysis (or code analysis) of the malware to extract features of the malware file. Their model consists of an input layer, an LSTM layer, a dropout layer, and a dense layer. A weakness of their system is the average accuracy of 90%.Agarap [52] proposed an SVM and CNN-based system for detecting malware. First, the binary string of the malware sample was transformed to 8-bit vectors, which are further processed and transformed to a grayscale image. Both multilayer perceptron and CNN were used for experiments and achieved better results with MLP; however, a below-average accuracy rate of about 80% was achieved.Other papers that focus on Windows-based malware detection include [53–64].


## 5. Android Malware Detection

Like Windows platform, in android platform, researchers have worked on malware and intrusion detection using deep learning. This section analyzes the research work performed over android platform, extracting meta-information highlighted in research questions (RQ7 and RQ8), in addition to the information considered in Windows-based malware detection literature. Summary of android-based malware detection techniques is shown in Table3.Devi [65] proposed a permission based android malware detection system. They extracted the manifest files and permissions from the android packages and generated feature vectors and trained their model using neural networks and k-means clustering algorithm. The weakness of this approach is comparatively lesser accuracy of 88%.Karbab et al. [66] proposed MalDozer, an android malware detection system based on the API method call sequence of the applications. MalDozer disassembles the classes.dex file of an android package to extract API method calls and discretize them by replacing each API method by an identifier and generates the semantic vectors. Then, they train the neural networks to predict android malware. The strength of their system is the high F1 score achieved on various datasets.Khedkar et al. [67] also proposed a permission-based android malware detection system. The FAST algorithm they designed use graph clustering method to cluster the features of the application and construct a trained dataset to classify new malware.Kim et al. [68] used multiple features for malware detection including API methods, op-code features, permission features, share library function op-code features, component features, and environment features to generate feature vectors for each feature. These vectors are then fed to the classification model to predict malware. The strength of their approach is the usage of multiple features unlike most of the others who used a single or two features.Milosevic and Huang [69] proposed a deep learning-based malware prediction system that uses CPU, memory, and battery usage to predict malware. Their unsupervised method is based on encoder-decoder and LSTM networks, using different applications to retrieve data, like CPU and battery usage. The weakness of their system is comparatively lower F1 score of about 80%.Yuan et al. [70] developed an online android malware detection system. The proposed system extracts three features, required permissions, sensitive API calls, and dynamic behavior and then use deep belief networks to detect malware in an application. Their deep learning model has two phases, an unsupervised learning phase and a supervised back propagation phase.Yuan et al. [71] proposed a deep learning-based method that includes extracting features like permissions, sensitive API calls, and dynamic behavior. They used more than 200 different features in their proposed framework and used deep belief networks for malware detection. Their claimed accuracy is over 96%.Yen and Sun [72] proposed a system that use the importance of words in apk file for malware detection. They extract the classes from apk file and convert them to java files and find the importance value of each word in the code. Then, they generate images by using the words importance from code using Term Frequency-Inverse Document Frequency (TF-IDF), a text mining and information retrieval method. These images were used to train and text their CNN based model.Xie et al. [73] proposed a CNN-based approach, which includes extracting seven different malware features: API calls, hardware features, filtered intent, requested permissions, used permission, and restricted API calls. The framework consists of Dataset Construction, which includes collecting samples, labelling and features extraction, and classification process in which feature vectors are transformed to matrices and the dataset is divided to training set and validation set. The strength of their system is the claimed accuracy of 99.25%.Wang et al. [74] combined deep autoencoder with a modified model of convolutional neural network they called CNN-S and proposed an android malware detection system that uses seven different features of applications to train their model. The features include restricted API calls, suspicious API calls, permissions, requested permissions, hardware features, filtered intents, and code-related patterns. The strength of their system is the claimed high accuracy rate of 99.82%.Luo et al. [75] proposed ITMF (image texture median filter) to analyze and detect android malware. Median filter is a filtering technique for removing or reducing noise from images and signals to improve processing and results. They obtain the malware binary file and convert it to a vector which is then transformed to grayscale image, which is then inputted to the ITMF. They extract features including API calls, used permissions, URL, and activity and train deep belief network for malware detection. They compared their model with shallow learning techniques and achieved better results with deep learning.Saif et al. [76] proposed a deep belief network-based android malware detection system. They used both static analysis and dynamic analysis of android application and extracted features like manifest components, API calls, dynamic behavior of the application, and system calls and generated feature vector. They applied relief feature selection by using relief algorithm, which outputs another vector with the quality measurement of features. This vector is inputted to the deep neural network.Pektaş and Acarman [77] proposed an android malware detection system that uses API calls graph. They build an API call graph for each execution path; the API call number is selected to generate graph embedding if it is equal to or greater than a threshold value and the graph embedding features are processed to be interpreted numerically. Then, the embedding vectors are inputted to the CNN-based deep learning model to classify malware.Pektaş and Acarman [78] built an android malware detection system that examines all the execution paths and detect malware by using features extracted from instruction call graph. Their method consists of pseudodynamic analysis of the application in which call graphs and execution paths are extracted in terms of op-codes. Then, they construct a flow graph for each execution path and process the graphs to be interpreted numerically and to generate vectors. The vectors are then inputted to the LSTM RNN-based deep learning model to determine the probability of being benign or malware. They compared their approach with traditional machine learning approaches and showed that deep learning achieves better accuracy rate.Nauman et al. [79] used different deep learning methods, including CNN, DBN, LSTM, and autoencoders on large-scale dataset for detecting android malware. They used the features from manifest file and those extracted through static analysis including requested permissions, components, filtered intents and restricted API calls, and so on as input to the deep learning model and evaluated the performance of different deep learning methods.Martín et al. [80] proposed a deep learning-based system CANDYMAN, which classifies malware by combining dynamic analysis and Markov chains. They use DroidBox tool to run the application and extract dynamic behavior. The information gathered include network data, read/write operations, services, loaded classes, file, and SMS services and permissions, which is reported in a JSON file. In the next step, the data from the JSON file is represented in terms of Markov chains. Finally, the Markov chains are transformed into feature vectors that are then fed into deep learning networks for malware classification. They performed experiments using different machine learning and deep learning algorithms; however, they achieved a lower accuracy of around 81%.Shiqi et al. [81] presented an attention-CNN-LSTM-based deep learning system for android malware detection. They use deep belief networks to extract texture fingerprint features and the malware activity embedding in vector space and then the malicious code is converted to grayscale image. The malware texture fingerprint features and the activity embedding in vector space are fed to the attention-CNN-LSTM-based deep learning model for malware classification. They compared their model to traditional machine learning algorithms and showed that they achieved a better accuracy with deep learning.Halim et al. [82] proposed an android malware detection system that uses Bag of Words (BOW) model to extract various features of the application, including hardware components, used permissions, requested permissions, application components, filtered intents, restricted API calls, suspicious API calls, and network addresses. They used two different deep learning models for malware detection, a CNN-LSTM model in which CNN is stacked over LSTM and an LSTM-CNN model in which LSTM is stacked on top of CNN. They achieved an accuracy of 96.76% with the CNN-LSTM model while 98.53% with LSTM-CNN model.Elsersy and Anuar [83] proposed a deep belief network-based deep learning system for android malware detection. They used Lasso features shrinkage and selection technique, which is used for features selection by means of absolute regularization penalty and evaluated traditional machine learning technique (K-NN classifier) and deep learning technique (DBN) for malware detection. They achieved better accuracy with deep learning technique; however, the accuracy rate they achieved was below average, 85.22%.D’Angelo et al. [84] generated sparse matrices from the sequence of the API calls to be used for malware detection. Their autoencoders based system represents the temporal behavior of the application by using the sequence of sparse matrices and extract features from the sparse matrices, which are then used to classify the application as malware or benign-ware.Chen et al. [85] proposed an android malware detection system that uses features like permissions and sensitive API calls extracted from the APK file. They model the features as a document and generate k-dimensional word vectors using word2vec. Finally, they use deep belief networks based deep learning system for malware classification.Amin et al. [86] proposed an android malware detection system based on various deep learning methods. They extract the .dex file (dalvik executable file) from the APK file and further use it to extract the byte code. This byte code is given as input to the deep learning model for training, feature engineering, and classification of the sample as malware or benign. They used different deep learning methods, including DAE, DBN, LSTM, BiLSTM, CNN, and RNN, and claimed to have achieved an accuracy of up to 99.9%.Alzaylaee et al. [87] proposed a malware detection system for android platform that runs the application to extract its features. They use DynaLog, a platform that runs a large number of android applications in sequence to log and extract dynamic features, such as API calls, actions, events, and permissions. They extract 178 features and rank them using InfoGain to select the top 120 of them for experiments. Other research articles that focused on android malware detection include [88–99].


## 6. IoT/IoBT Malware Detection

A number of studies focused on malware detection in the Internet of Things and Internet of Battlefield (Military) Things (IoBT/IoMT) environments. These studies are analyzed and summarized in the following section.Azmoodeh et al. [100] proposed a two-phase method for malware detection. They first generate the Op-code sequence graph by using the selected features and then use deep eigenspace learning to classify Internet-of-things and Internet of (battlefield) things (IoBT) malware. The strength of their system is the claimed accuracy of over 99%.Xiao et al. [101] also combined machine learning and deep learning, combining DT, NB, SVM, and KNN with autoencoders. They proposed a behavior-based deep learning framework for malware detection in the Internet of Things environment. Their proposed model consists of IoT environment, which includes local computers and smart devices and a cloud platform module (CP module). CP provides storage space, constructs behavior graph, and transforms the API call graphs to binary vectors. These vectors are used as input to the stacked autoencoders-based deep learning model.Ullah et al. [102] proposed a convolutional neural network-based system for detecting pirated software applications and files infected by malware in the IoT environment. Their system consists of a preprocessing module that transforms the malware binary file to grayscale image, a convolutional neural network to which the training images are inputted so that the classifier identifies the respective malware families using the images, a convolution layer that is used to extract meaningful features, and a pooling layer that is used to minimize the consequences of image distortion and increase CNN functioning. They claimed to have a better accuracy rate of 96% (piracy detection) and 97.46% (malware detection) as compared to other traditional machine learning techniques.Haddadpajouh et al. [103] proposed an LSTM-based system for detecting malware in the Internet of things environment. They first collected samples of malware and benignware and decompiled them using object-dump tool. They used a Linux hash script to extract op-codes from the samples and used text mining techniques to generate features from the op-codes. Next, they used LSTM network with two hidden networks for malware detection. They compared their approach with several traditional learning methods and showed that they achieved much better accuracy rate with deep learning.Al-Hawawreh et al. [104] proposed a deep autoencoder and deep feed-forward neural network-based system for detecting malicious activities in IoT environment. The deep autoencoder-based model learns to use normal network observations and creates initialization parameters and learns the representation of normal behavior. The parameters created at this stage are used as input to the DFFNN-based model for detecting new attacks.Abusnaina et al. [105] used graph embedding to classify malicious programs in the IoT environment.Naeem et al. [106] extracted bytecode from the java class file of the malicious software and used this bytecode for detecting malware in the Internet of things environment.


## 7. Other Platforms

Some of the studies focused on malware in the cloud environment, web applications or did not even mention what operating system or platform they targeted, or targeted multiple platforms at a time. These studies are analyzed in the following section and summarized in Table5.Lu et al. [107] constructed their own deep neural network for malware classification, which they named MalDeepNet. They also used several features, like API features, PID features, RET features, EXINFO features, and reboot features, and implemented TB-MalNet (Text-based MalNet) and IB-MalNet (Image-based MalNet) for malware prediction.Yu et al. [108] investigated the use of deep learning algorithms for Domain Generation Algorithms (DGAs) and developed an LSTM-based deep learning model for DGA detection trained with weakly labelled data obtained from real traffic. They achieved an accuracy of around 98% on different datasets.Vinayakumar et al. [109] proposed a deep learning-based distributed system for detecting cyber-attacks. The proposed system was built using big data processing frameworks Apache Hadoop YARN and Apache Spark. They used various shallow learning algorithms, like NB, RF, SVM, LR, KNN, and so on, and deep neural networks with different layers and evaluated their performance on different datasets and experimentally shown that deep neural networks perform better than traditional machine learning algorithms.Song et al. [110] proposed an intrusion detection system for the controller area network in vehicles to protect the CAN bus of the vehicle. Their CNN-based system retrieves CAN IDs from the logged CAN and assembles data frames, each consisting of 29 sequential IDs. The data frames are then processed and classified as attack or nonattack. They also compared their proposed system with traditional machine learning techniques and experimentally showed that DL performed better. The strength of their system is the claimed F1 score of above 99%.Priyadarshini and Barik [111] proposed a DDoS defense system that is capable of detecting and mitigating denial of service attacks in fog and cloud computing environment. They use Hogzilla dataset to train their LSTM-based deep learning model for DDoS defense and achieve a high accuracy rate of 98.88%.Pektaş and Acarman [112] presented a hybrid, RNN- and CNN-based, deep learning system for detecting botnet. They extract flow features from the network traffic and transform them to multidimensional feature vector. The feature vector is inputted to the classification model for detecting whether the element is normal or botnet. They use the connection patterns created due to the data transmission between botnets and servers and split network traffic between endpoints and represent them as graph to extract features. The strength of their system is the high accuracy rate of nearly 99.4% on average.Pan et al. [113] proposed a deep learning-based system for detecting attack in the web traffic by analyzing web applications. Their system uses robust software modeling tool (RSMT), which is a tool that targets languages that run on JVM, and extracts traces of program execution and generates models of behavior of the running application. RSMT captures features that represent program behavior, which are used as input to the Stacked Autoencoders-based deep learning model for detecting anomalies in web applications.Loukas et al. [114] used deep multilayer perceptron and recurrent neural networks and built an intrusion detection system in the cloud environment. They used a robotic vehicle to evaluate their system by detecting various type attacks, including denial-of-service attack, command injection attack, and malware attack. They tested various traditional learning models and achieved much better average accuracy of 87% with deep learning.Jeong et al. [115] proposed a system for detecting malware in PDF files. Their CNN-based model consists of one embedding layer, two convolutional layers, one pooling layer, one fully connected layer, and one output layer. The first layer is used to represent contextual meaning of the byte values and generate E-dimensional vectors, which are then given as input to the convolutional layers.Homayoun et al. [116] proposed an LSTM and CNN-based deep learning approach for ransomware detection and classification. They use a deep feature extractor and a one-class classifier. It records the executed events when an application is started and transforms the sequence of the events to a numerical form and combines the input datasets to a single dataset. They use two different deep learning tasks for ransomware detection and classification, respectively. Other literature focusing on different types of malware and vulnerabilities detection include [117–119].


## 8. Research during the Recent Years

As obvious from Figure2, the use of deep learning methods for malware and intrusion detection system is on the rise and has been increasing each year. In this section, we have selected a few important and most cited studies from the recent years (i.e., 2021–22) that the readers and researchers would be more interested in. Table 6 summarizes these studies.


## 9. Major Deep Learning Algorithms Used in Malware Detection

In order to answer RQ4 (What are the major DL algorithms used in the domain of malware detection?), we collected information about the usage of different DL algorithms in any form by the researchers. From the summary results shown in Figure4, it is evident that Convolutional Neural Networks were used in most of the studies by the researchers which represents more than 50% of the publications surveyed, while LSTM-based neural networks in different forms were used by 25 researchers, which make up 25%. Similarly, DBN-based and AE-based algorithms were used in 13 and 11 publications that form 12% and 10.3%, respectively, of the total publications reviewed. Like many other domains, in malware detection and classification, most of the researchers have preferred convolutional neural networks when choosing a deep learning model. CNN is one of the most popular deep learning networks, which is capable of detecting the significant features without supervision. It is widely used for classification tasks, such as plant diseases, object detection, medical image analysis and computer vision, and so on. It is especially reported effective in image classification and image/object detection.Figure 4
Major deep learning algorithms used in the domain of malware detection.


## 10. Discussion

### 10.1. Effectiveness of Deep Learning in Malware Detection

Deep learning produces best results with unstructured data. As most of the data produced by various systems is unstructured and in various formats, we either need to structure the data or have systems that have the capability to process unstructured data. Deep learning enables us to develop malware detection systems that can produce better results with unstructured and unlabeled data as well. Moreover, a deep learning algorithm can perform thousands of complex and repetitive tasks in very short time when trained once and produces accurate results as long as the raw data provided represents the problem.Traditional malware detection techniques do not use machine learning or deep learning algorithms, and their performance is quite limited when it comes to detecting new types of malware. They rely on regularly updating their “malware definitions,” which are used to detect threats. On the other hand, machine learning and deep learning-based algorithms can discover complex structures in structured and unstructured data when once trained and are very useful in developing effective malware detection systems. Hackers are developing malware that can change their code when propagated and thus hard to detect with the traditional pattern matching techniques. These malware can also deceive the traditional pattern matching-based systems easily in an intelligent manner. The different behavioral patterns that malware share could be used to detect unknown malware using ML and DL techniques.


### 10.2. Performance of DL Compared with Traditional Learning

Deep learning algorithms have become popular as they can deliver more accurate results when trained with large amounts of data as compared to traditional learning algorithms. These algorithms can learn high-level features from data and mostly do not need domain expertise and hard-core feature extraction. The authors of some of the papers we reviewed used both deep learning and traditional learning algorithms for malware detection and experimentally showed that deep learning algorithms performed better than machine learning algorithms. Rhode et al. [39] proposed an early-stage malware detection system that is intended to detect malware within a few seconds of execution of a program. They used RNN and compared it with traditional learning algorithms, such as SVM. SVM achieved considerable accuracy of 80%, but RNN outperformed it after 1 second of execution and achieved an accuracy of 96% at 19 seconds into execution. Random Forest classifier achieved accuracy of 92% while Decision Trees achieved 92.6% accuracy. Haddadpajouh et al. [103] achieved an accuracy of 98.18% using RNN-LSTM for threat hunting in the Internet of things environment. They also used traditional machine learning algorithms and achieved the highest accuracy of 94% using KNN. Loukas et al. [114] developed an intrusion detection system for detecting cyber-attacks against vehicles, which achieved an accuracy of 86.9% with RNN. They also used several machine learning algorithms, including Logistic Regression, DT, RF, and SVM, achieving accuracy of 73.3%, 74%, 77.3%, and 79.9%, respectively. The cyber security threats detection system developed by Ullah et al. [102] performed better compared to other systems that used traditional learning algorithms and achieved much higher accuracy of 96%. Vinayakumar et al. [109] developed a deep neural network-based intrusion detection system and achieved an accuracy of up to 99.2%. They achieved a much lower average accuracy of around 80% with classical machine learning algorithms. Luo et al. [81] worked on detecting android malware and used Attention-CNN-LSTM in their system. They compared their deep learning-based model with SVM-based and KNN-based models and achieved a higher average accuracy of 96% compared to 95% with KNN and 94% with SVM. Similarly, Pektaş and Acarman [78] proposed an android malware detection system using instruction calls graphs and achieved 91.4% accuracy. They compared the proposed method with traditional learning algorithms, including KNN, Logistic Regression, SVN, and RF, and achieved accuracy of 80%, 70%, 79%, and 89%, respectively. Schranko de Oliveira and Sassi [90] acheived an accuracy of 91% with their deep neural network-based android malware detection system outperforming several ML algorithms, including SVM, RF, Logistic Regression, Extra Trees, and KNN. On the contrary, Jain et al. [55] achieved better accuracy of 97.7% using ELM with just one hidden layer as compared to 96.3% with CNN based architecture. Similarly, Pastor et al. [118] tested various traditional learning algorithms and CNN for detecting cryptomining traffic and achieved equal or better results with traditional learning algorithms.In most of the cases, deep learning models performed much better than traditional learning methods. All these statistics show that deep learning algorithms are more capable of detecting and hunting malware or other threats and using shallow learning techniques may not lead us to a scalable solution with significant accuracy. However, it is not guaranteed that DL algorithms will always perform better than ML algorithms as some of the studies and our experiment’s results depict. In our case, we compared the performance of Deep Autoencoders with different ML algorithms and achieved a higher accuracy rate with traditional learning models. Table11 summarizes the results of our experiments. Our AE-based model could outperform only the Naïve Bayes ML algorithm.Table 11
Analysis of the sustainability of android-based malware detection techniques.
Ref.Evolvability/ability of identifying new malware? (yes/no)DL model need updating/retraining? (yes/no)Sustainability/resilience against evolution? (yes/no)Sustainable up to years/accuracyInitial accuracy/F1 scoreTowards sustainable android malware detection [132]YesAfter 5 yearsYes5 (82%)Above 93%MaMaDroid [133]YesAfter 2 yearsYes2 (87%)99%Frequency analysis model (FAM)—a variant of MaMaDroid [134]YesAfter several yearsYesSeveral (76%)81%DroidSpan [135]YesAfter 7 yearsYes1–7 (21–37% superior than competitor)6–32% superior than competitorRevealDroid [136]YesAfter 3 yearsYes3 (87%)98%DroidCat [137]YesAfter 9 yearsYes9 (97%)97%G-Droid [138]Yes—Yes—98.99%


### 10.3. Challenges in Malware Detection Using Deep Learning

#### 10.3.1. The Issue of Data Unavailability

Machine learning and deep learning algorithms need a huge amount of training data to start with. In malware and threat detection, one of the biggest challenges is to provide a sufficient amount of malware and benign samples to the algorithm. Many datasets are available for use in research, but they need to be updated frequently so that the most recent malware samples could be used for training models.


#### 10.3.2. The Issue of Data Noise and Model Overfitting

Another big challenge is the risk of wrongly labelled and noisy data, which may result in “overtraining the model” that leads to incorrect results.


#### 10.3.3. The Issue of Model Validation and Response to New Threats

Many of the studies achieved high accuracy rates. However, they did not provide experimental results demonstrating how their systems would perform if a new type of malware attacked. New types of malware are being created across the world with passage of time and the malware detection systems should not only be able to detect variants of the malware samples that are used for training but also new types of malware to actually get the advantage of deep learning over the traditional threat detection systems.


#### 10.3.4. The Issue of Evolution

Since a key issue in the android ecosystem is its fast evolution and various problems caused by the evolution [139, 140], the development of a flexible model that could be used at all times for the detection of new types of threats in the future is required. In the case the model does not need frequent retraining to cope with the situation of arising new malware but needs to update the model after a few years with very low degradation in the model performance, an evolvable model leads to sustainability in the model.


#### 10.3.5. The Issue of Automatic Selection of DL Algorithm

In literature, a large number of DL methods can be found, which deal with complex problems, like malware classification over the big data. However, like ML algorithm (s) selection problem [141], the researchers always find it difficult to decide which method to pick for their problems in hand without frequently training, testing, and adopting the model.


#### 10.3.6. The Issue of Sustainability

The frequent changes and continuous evolution of android malware demands for frequent retaining of the supervised malware detection models, which is a challenging job [142–144]. This requires building a sustainable malware detection model to update itself over the time in an effective and scalable manner. In case of declaring a model as sustainable, the frequency of retaining, duration for which the model is sustainable, and degradation in performance after the declared period are a few characteristics that need to be considered. For further details, Table 11 summarizes a few sustainable models with their key characteristics.


#### 10.3.7. The Issue of Automatic Features Engineering and Analysis for Robust Modeling

One of the key challenges is how to pick or automatically learn, in the case of using deep learning for automatic feature engineering, features that stand the best of the time and future without frequent retraining. In literature, static, dynamic, and hybrid analysis methods have been used which automatically extract features for learning the DL model [145–147]. Column 2, “Automatic DL algorithm selection (yes/no),” of Table 3 summarizes these methods.


### 10.4. Data Quality for Malware Detection Using Deep Learning

Data quality is an essential component for machine and deep learning tools used for malware detection. Hence, along with technical approaches, availability of a sizable and informative dataset plays an essential role in the predictive accuracy of such systems. Therefore, the quality of dataset should be carefully considered when creating and developing predictive models and tools for malware detection [148]. Such datasets are created by the research community to serve as a source of research for empirical analysis and extracting new insights about apps. In case of malware detection, data quality may be read as the number and types of apps used in android-based malware detection and operating system’s application programming interfaces (API) calls in case of Windows-based malware detection [149]. The details are explained in the subsequent sections.


### 10.5. Android-Based Malware Detection Datasets

In literature, a number of android-based malware datasets have been curated and made publicly available for researchers to perform their research activities. The details of a few most important datasets are made available here, in Table12, with brief description of their characteristics.Table 12
Android-based malware detection datasets for research community.
ReferenceDataset: description, size, type[150]15,451 benign apps and 15,183 malwareAndroZoo [151]More than three million appsAndroCT [152]A large-scale dataset on the run-time traces of function calls in 35,974 benign and malicious android apps from ten historical years (2010 through 2019)Rmvdroid [153]Malware dataset containing 9,133 samples that belong to 56 malware families over the four years of 2014–18[154]17,664 apps sampled from the apps developed in each of the past eight years (2012–21)AndroZooOpen [155]AndroZooOpen, currently contains over 45,000 app artefacts, a representative picture of Github-hosted android appsDeep ground [156]Dataset (containing 24,650 malware apps)DREBINIn an evaluation with 123,453 applications and 5,560 malware samples DREBINMalgenome [157]1,200 malware samples that cover the majority of existing android malware families, ranging from their debut in August 2010 to recent ones in October 2011


### 10.6. Windows-Based Malware Datasets

Like android-based malware detections datasets, different malicious programs belong to different families, such as backdoor, adware, downloader, dropper, spyware, trojan, virus and worms, and so on, have been studied by researchers in windows environment in the form of API calls. These are collected into different datasets, which can be used by analysts to build intelligent automatic malware detection systems in windows environment. A few examples of such datasets have been provided in Table13.Table 13
Windows-based malware datasets.
ReferenceDataset: description, size, typeAPI Call dataset [149]7107 different malicious software belonging to various families, such as virus, backdoor, trojan, and so on, have been analyzed, categorized into its different families, and made available for researchers to work on.EMBER [158]A labelled benchmark dataset for training machine learning models to statically detect malicious Windows portable executable files. This dataset includes features extracted from 1.1 M binary files.SOREL-20 M [159]A large-scale dataset consisting of nearly 20 million files with preextracted features and metadata, high-quality labels derived from multiple sources, information about vendor detections of the malware samples at the time of collection, and additional “tags” related to each malware sample to serve as additional targets.


## 10.1. Effectiveness of Deep Learning in Malware Detection

Deep learning produces best results with unstructured data. As most of the data produced by various systems is unstructured and in various formats, we either need to structure the data or have systems that have the capability to process unstructured data. Deep learning enables us to develop malware detection systems that can produce better results with unstructured and unlabeled data as well. Moreover, a deep learning algorithm can perform thousands of complex and repetitive tasks in very short time when trained once and produces accurate results as long as the raw data provided represents the problem.Traditional malware detection techniques do not use machine learning or deep learning algorithms, and their performance is quite limited when it comes to detecting new types of malware. They rely on regularly updating their “malware definitions,” which are used to detect threats. On the other hand, machine learning and deep learning-based algorithms can discover complex structures in structured and unstructured data when once trained and are very useful in developing effective malware detection systems. Hackers are developing malware that can change their code when propagated and thus hard to detect with the traditional pattern matching techniques. These malware can also deceive the traditional pattern matching-based systems easily in an intelligent manner. The different behavioral patterns that malware share could be used to detect unknown malware using ML and DL techniques.


## 10.2. Performance of DL Compared with Traditional Learning

Deep learning algorithms have become popular as they can deliver more accurate results when trained with large amounts of data as compared to traditional learning algorithms. These algorithms can learn high-level features from data and mostly do not need domain expertise and hard-core feature extraction. The authors of some of the papers we reviewed used both deep learning and traditional learning algorithms for malware detection and experimentally showed that deep learning algorithms performed better than machine learning algorithms. Rhode et al. [39] proposed an early-stage malware detection system that is intended to detect malware within a few seconds of execution of a program. They used RNN and compared it with traditional learning algorithms, such as SVM. SVM achieved considerable accuracy of 80%, but RNN outperformed it after 1 second of execution and achieved an accuracy of 96% at 19 seconds into execution. Random Forest classifier achieved accuracy of 92% while Decision Trees achieved 92.6% accuracy. Haddadpajouh et al. [103] achieved an accuracy of 98.18% using RNN-LSTM for threat hunting in the Internet of things environment. They also used traditional machine learning algorithms and achieved the highest accuracy of 94% using KNN. Loukas et al. [114] developed an intrusion detection system for detecting cyber-attacks against vehicles, which achieved an accuracy of 86.9% with RNN. They also used several machine learning algorithms, including Logistic Regression, DT, RF, and SVM, achieving accuracy of 73.3%, 74%, 77.3%, and 79.9%, respectively. The cyber security threats detection system developed by Ullah et al. [102] performed better compared to other systems that used traditional learning algorithms and achieved much higher accuracy of 96%. Vinayakumar et al. [109] developed a deep neural network-based intrusion detection system and achieved an accuracy of up to 99.2%. They achieved a much lower average accuracy of around 80% with classical machine learning algorithms. Luo et al. [81] worked on detecting android malware and used Attention-CNN-LSTM in their system. They compared their deep learning-based model with SVM-based and KNN-based models and achieved a higher average accuracy of 96% compared to 95% with KNN and 94% with SVM. Similarly, Pektaş and Acarman [78] proposed an android malware detection system using instruction calls graphs and achieved 91.4% accuracy. They compared the proposed method with traditional learning algorithms, including KNN, Logistic Regression, SVN, and RF, and achieved accuracy of 80%, 70%, 79%, and 89%, respectively. Schranko de Oliveira and Sassi [90] acheived an accuracy of 91% with their deep neural network-based android malware detection system outperforming several ML algorithms, including SVM, RF, Logistic Regression, Extra Trees, and KNN. On the contrary, Jain et al. [55] achieved better accuracy of 97.7% using ELM with just one hidden layer as compared to 96.3% with CNN based architecture. Similarly, Pastor et al. [118] tested various traditional learning algorithms and CNN for detecting cryptomining traffic and achieved equal or better results with traditional learning algorithms.In most of the cases, deep learning models performed much better than traditional learning methods. All these statistics show that deep learning algorithms are more capable of detecting and hunting malware or other threats and using shallow learning techniques may not lead us to a scalable solution with significant accuracy. However, it is not guaranteed that DL algorithms will always perform better than ML algorithms as some of the studies and our experiment’s results depict. In our case, we compared the performance of Deep Autoencoders with different ML algorithms and achieved a higher accuracy rate with traditional learning models. Table11 summarizes the results of our experiments. Our AE-based model could outperform only the Naïve Bayes ML algorithm.Table 11
Analysis of the sustainability of android-based malware detection techniques.
Ref.Evolvability/ability of identifying new malware? (yes/no)DL model need updating/retraining? (yes/no)Sustainability/resilience against evolution? (yes/no)Sustainable up to years/accuracyInitial accuracy/F1 scoreTowards sustainable android malware detection [132]YesAfter 5 yearsYes5 (82%)Above 93%MaMaDroid [133]YesAfter 2 yearsYes2 (87%)99%Frequency analysis model (FAM)—a variant of MaMaDroid [134]YesAfter several yearsYesSeveral (76%)81%DroidSpan [135]YesAfter 7 yearsYes1–7 (21–37% superior than competitor)6–32% superior than competitorRevealDroid [136]YesAfter 3 yearsYes3 (87%)98%DroidCat [137]YesAfter 9 yearsYes9 (97%)97%G-Droid [138]Yes—Yes—98.99%


## 10.3. Challenges in Malware Detection Using Deep Learning

### 10.3.1. The Issue of Data Unavailability

Machine learning and deep learning algorithms need a huge amount of training data to start with. In malware and threat detection, one of the biggest challenges is to provide a sufficient amount of malware and benign samples to the algorithm. Many datasets are available for use in research, but they need to be updated frequently so that the most recent malware samples could be used for training models.


### 10.3.2. The Issue of Data Noise and Model Overfitting

Another big challenge is the risk of wrongly labelled and noisy data, which may result in “overtraining the model” that leads to incorrect results.


### 10.3.3. The Issue of Model Validation and Response to New Threats

Many of the studies achieved high accuracy rates. However, they did not provide experimental results demonstrating how their systems would perform if a new type of malware attacked. New types of malware are being created across the world with passage of time and the malware detection systems should not only be able to detect variants of the malware samples that are used for training but also new types of malware to actually get the advantage of deep learning over the traditional threat detection systems.


### 10.3.4. The Issue of Evolution

Since a key issue in the android ecosystem is its fast evolution and various problems caused by the evolution [139, 140], the development of a flexible model that could be used at all times for the detection of new types of threats in the future is required. In the case the model does not need frequent retraining to cope with the situation of arising new malware but needs to update the model after a few years with very low degradation in the model performance, an evolvable model leads to sustainability in the model.


### 10.3.5. The Issue of Automatic Selection of DL Algorithm

In literature, a large number of DL methods can be found, which deal with complex problems, like malware classification over the big data. However, like ML algorithm (s) selection problem [141], the researchers always find it difficult to decide which method to pick for their problems in hand without frequently training, testing, and adopting the model.


### 10.3.6. The Issue of Sustainability

The frequent changes and continuous evolution of android malware demands for frequent retaining of the supervised malware detection models, which is a challenging job [142–144]. This requires building a sustainable malware detection model to update itself over the time in an effective and scalable manner. In case of declaring a model as sustainable, the frequency of retaining, duration for which the model is sustainable, and degradation in performance after the declared period are a few characteristics that need to be considered. For further details, Table 11 summarizes a few sustainable models with their key characteristics.


### 10.3.7. The Issue of Automatic Features Engineering and Analysis for Robust Modeling

One of the key challenges is how to pick or automatically learn, in the case of using deep learning for automatic feature engineering, features that stand the best of the time and future without frequent retraining. In literature, static, dynamic, and hybrid analysis methods have been used which automatically extract features for learning the DL model [145–147]. Column 2, “Automatic DL algorithm selection (yes/no),” of Table 3 summarizes these methods.


## 10.3.1. The Issue of Data Unavailability

Machine learning and deep learning algorithms need a huge amount of training data to start with. In malware and threat detection, one of the biggest challenges is to provide a sufficient amount of malware and benign samples to the algorithm. Many datasets are available for use in research, but they need to be updated frequently so that the most recent malware samples could be used for training models.


## 10.3.2. The Issue of Data Noise and Model Overfitting

Another big challenge is the risk of wrongly labelled and noisy data, which may result in “overtraining the model” that leads to incorrect results.


## 10.3.3. The Issue of Model Validation and Response to New Threats

Many of the studies achieved high accuracy rates. However, they did not provide experimental results demonstrating how their systems would perform if a new type of malware attacked. New types of malware are being created across the world with passage of time and the malware detection systems should not only be able to detect variants of the malware samples that are used for training but also new types of malware to actually get the advantage of deep learning over the traditional threat detection systems.


## 10.3.4. The Issue of Evolution

Since a key issue in the android ecosystem is its fast evolution and various problems caused by the evolution [139, 140], the development of a flexible model that could be used at all times for the detection of new types of threats in the future is required. In the case the model does not need frequent retraining to cope with the situation of arising new malware but needs to update the model after a few years with very low degradation in the model performance, an evolvable model leads to sustainability in the model.


## 10.3.5. The Issue of Automatic Selection of DL Algorithm

In literature, a large number of DL methods can be found, which deal with complex problems, like malware classification over the big data. However, like ML algorithm (s) selection problem [141], the researchers always find it difficult to decide which method to pick for their problems in hand without frequently training, testing, and adopting the model.


## 10.3.6. The Issue of Sustainability

The frequent changes and continuous evolution of android malware demands for frequent retaining of the supervised malware detection models, which is a challenging job [142–144]. This requires building a sustainable malware detection model to update itself over the time in an effective and scalable manner. In case of declaring a model as sustainable, the frequency of retaining, duration for which the model is sustainable, and degradation in performance after the declared period are a few characteristics that need to be considered. For further details, Table 11 summarizes a few sustainable models with their key characteristics.


## 10.3.7. The Issue of Automatic Features Engineering and Analysis for Robust Modeling

One of the key challenges is how to pick or automatically learn, in the case of using deep learning for automatic feature engineering, features that stand the best of the time and future without frequent retraining. In literature, static, dynamic, and hybrid analysis methods have been used which automatically extract features for learning the DL model [145–147]. Column 2, “Automatic DL algorithm selection (yes/no),” of Table 3 summarizes these methods.


## 10.4. Data Quality for Malware Detection Using Deep Learning

Data quality is an essential component for machine and deep learning tools used for malware detection. Hence, along with technical approaches, availability of a sizable and informative dataset plays an essential role in the predictive accuracy of such systems. Therefore, the quality of dataset should be carefully considered when creating and developing predictive models and tools for malware detection [148]. Such datasets are created by the research community to serve as a source of research for empirical analysis and extracting new insights about apps. In case of malware detection, data quality may be read as the number and types of apps used in android-based malware detection and operating system’s application programming interfaces (API) calls in case of Windows-based malware detection [149]. The details are explained in the subsequent sections.


## 10.5. Android-Based Malware Detection Datasets

In literature, a number of android-based malware datasets have been curated and made publicly available for researchers to perform their research activities. The details of a few most important datasets are made available here, in Table12, with brief description of their characteristics.Table 12
Android-based malware detection datasets for research community.
ReferenceDataset: description, size, type[150]15,451 benign apps and 15,183 malwareAndroZoo [151]More than three million appsAndroCT [152]A large-scale dataset on the run-time traces of function calls in 35,974 benign and malicious android apps from ten historical years (2010 through 2019)Rmvdroid [153]Malware dataset containing 9,133 samples that belong to 56 malware families over the four years of 2014–18[154]17,664 apps sampled from the apps developed in each of the past eight years (2012–21)AndroZooOpen [155]AndroZooOpen, currently contains over 45,000 app artefacts, a representative picture of Github-hosted android appsDeep ground [156]Dataset (containing 24,650 malware apps)DREBINIn an evaluation with 123,453 applications and 5,560 malware samples DREBINMalgenome [157]1,200 malware samples that cover the majority of existing android malware families, ranging from their debut in August 2010 to recent ones in October 2011


## 10.6. Windows-Based Malware Datasets

Like android-based malware detections datasets, different malicious programs belong to different families, such as backdoor, adware, downloader, dropper, spyware, trojan, virus and worms, and so on, have been studied by researchers in windows environment in the form of API calls. These are collected into different datasets, which can be used by analysts to build intelligent automatic malware detection systems in windows environment. A few examples of such datasets have been provided in Table13.Table 13
Windows-based malware datasets.
ReferenceDataset: description, size, typeAPI Call dataset [149]7107 different malicious software belonging to various families, such as virus, backdoor, trojan, and so on, have been analyzed, categorized into its different families, and made available for researchers to work on.EMBER [158]A labelled benchmark dataset for training machine learning models to statically detect malicious Windows portable executable files. This dataset includes features extracted from 1.1 M binary files.SOREL-20 M [159]A large-scale dataset consisting of nearly 20 million files with preextracted features and metadata, high-quality labels derived from multiple sources, information about vendor detections of the malware samples at the time of collection, and additional “tags” related to each malware sample to serve as additional targets.


## 11. Recommendations and Future Perspective

In this section, on the basis of our study and observations, we make some recommendations for the readers and the future researchers in the domain of security and malware threat detection using ML and DL techniques. A large number of researchers have focused on developing intelligent systems for malware threat detection and classification; however, very few of them have considered using big data analytics tools.(i)
With the growth of the Internet, the enormous amount of data being generated could not be handled using traditional data processing techniques. Big data analytics frameworks, such as Apache Hadoop and Apache Spark, are being adopted by organizations and websites to handle the big data generated in an efficient manner in relatively lesser time, which would not be possible otherwise.(ii)
The same is the case with security systems and threat hunting software, which need to deal with huge amounts of data within the machines and across the web. DL- and ML-based systems, integrated with big data processing tools, may be much more efficient and cost effective, especially in the domain of Internet security.(iii)
It is a big question whether the different DL-based security systems proposed in these studies would perform as good with big data as they perform theoretically.(iv)
We also observe that the deep learning-based security systems have mostly focused on windows and android platforms as compared to web-based security systems. Internet security is not less important than securing a computer offline or a smartphone.(v)
The results of this study lead us to the conclusion that developing deep learning-based intelligent Internet security systems is one of the areas in DL we need to focus on.(vi)
As obvious from the results of this study, many researchers have achieved a very high accuracy rate of up to 99.9% in malware detection; however, we need to have these malware hunting systems operating in real world performing as perfectly as these statistics show. Future researchers should focus on how to enable the users to easily and effectively use deep learning for protection against malware.(vii)
The scholars are recommended to work on sustainable and self-evolvable models that do not require frequent retraining.


## 12. Conclusion

In this review paper, we extensively studied the recent research publications that aimed at using deep learning for malware detection on various platforms, like Windows, smartphones, IoT, and the Internet. We searched five different libraries, including Google Scholar, Springer, Science Direct, IEEE Explore, and ACM Digital Library, to retrieve the relevant literature published during the last six years. We collected a total of 290 studies and then carefully studied all of them to select the studies to include in this survey. We excluded duplicates, non-English literature, book chapters, SLRs, and conference papers and finally selected a total of 107 publications to review. A lot of work has been done in the field of DL techniques for malware detection and classification, and various systems have been developed that have achieved accuracy as high as 99.9%. The statistics show that CNN, AE, RNN, and LSTM are used by most of the researchers. Python and Python-based libraries, like TensorFlow, Keras, and scikit-learn, are widely used to implement the DL models. However, there is less or no information about how these DL-based security systems would perform when applied in real-world scenarios and the question remains unanswered whether these systems would be able to handle the huge amounts of data being produced in organizations and online.The current study presents a big picture of the deep learning-based models used for threats classification and detection on a number of platforms, including Windows, Android, IoT, cloud computing, and the Internet. In the future, we intend to conduct extensive reviews for each of these platforms that will be useful for the researchers focusing on a specific platform.



---

*Source: 2959222-2022-10-10.xml*