# Methods to Identify the Family of Advanced Persistent Threats Based on Deep Neural Network and n-gram of API calls

Mohammad Ali Shafiee, Mehdi Shajari, Mahdiyeh Barzegar

► To cite this version:

Mohammad Ali Shafiee, Mehdi Shajari, Mahdiyeh Barzegar. Methods to Identify the Family of Advanced Persistent Threats Based on Deep Neural Network and n-gram of API calls. 2023. hal-03966915

HAL Id: hal-03966915

https://hal.science/hal-03966915

Preprint submitted on 8 Feb 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Methods to Identify the Family of Advanced
Persistent Threats Based on Deep Neural Network
and n-gram of API calls

Mohammad Ali Shafiee

Computer Engineering Department
Amirkabir University of Technology
Tehran, Iran

m.ali.shafiee@aut.ac.ir

Mehdi Shajari

Information Technology Management
Toronto Metropolitan University
Toronto, Canada

mehdi.shajari@torontomu.ca

Mahdiyeh Barzegar

Faculty of Computer Science
Free University of Bozen-Bolzano
Bolzano, Italy

mbarzegar@unibz.it

**Abstract**—Advanced Persistent Threat attacks (APTs) have a set of special features that make them different from other attacks. They are stealth, target-based, and designed by expert teams. There are various methods to detect malwares, but since the APT attacks are complex and well-designed to evade detection with a minimum fingerprint on the target systems, they are difficult to detect. Research on APTs has two challenges: the complexity of the attack and the small number of identified samples. These challenges have some drawbacks to the accuracy of malware classifiers. We address these two challenges in this study by presenting two methods. We used static and dynamic malware analysis as an input to the deep neural network to address the complexity of the attack challenge and n-gram of API calls as an input to four machine learning algorithms to address the small number of available samples. The feature selection phase of these two methods employs the TF-IDF measure to identify valuable terms in the malware analysis report. The proposed hybrid approach shows better results for the dataset we used for this research compared to the other baseline methods. In this research, we collected an APT dataset with a clear validation method which will benefit our future research.

**Index Terms**—Advanced Persistent Threat, Malware, Classification, Deep Neural Network, Machine Learning, Attack Detection

# I. INTRODUCTION

Advanced Persistent Threat (APT) is the most complex and sophisticated type of cybersecurity attack and on the rise in recent years. It is a set of stealthy and continuous processes targeting a specific entity that is highly valuable for the attackers, e.g., the government, financial industry, energy industry, or military. The attack agent is a highly skilled team with adequate financial and technical support. These attacks start from one point and then slowly move (i.e., lateral movement) in the organization over a long period of time to reach the desired goal. In different steps of this attack, a variety of unknown and complex malwares are used. Since these attacks can cause significant damage to the target organization and endanger its operation, it is crucial to detect and defend them before they reach their goals. As these malwares are target based and well-designed by expert teams to evade the detection systems, it is difficult to detect them by general

malware detection engines. Thus, a special malware classifier is needed to identify various APT attack categories.

Due to the privacy protection mandate of the organizations and the classified information, the victim organizations did not disclose much information about the attacks and the malware samples obtained. Therefore, the problem of APT malware detection has the complexity of the relations between features and a small number of detected samples. So, it is obvious that the researchers have limitations in this field to develop an effective solution.

In this paper we introduce two methods based on the TF-IDF measure to extract the most valuable terms in the malware analysis reports. The first method emphasizes the complexity of the relations between features based on a deep neural network. This method uses static and dynamic malware analysis reports as hybrid features to detect malwares. In the second method, we provided an effective way to detect malware despite the lack of a huge APT samples dataset based on machine learning algorithms using n-gram of API calls. Also, we collected a set of APT samples in a dataset for further research on this topic. The main contributions of this paper are as follows:

* Proposing an improved multi-class APT family classifier that can predict the family label of each sample with a high accuracy using deep neural network and machine learning algorithms.

* Using Deep Neural Networks and N-gram of API calls with machine learning algorithms to evaluate the best approach that can be improved by parameters and increase in number of samples for APT classification.

* Representing the feature vectors extracted from the malware analysis report using the TF-IDF measure (borrowed from information retrieval discipline).

* Extracting other features than families like the type of target system by transfer learning which can show the high-level features abstraction of the APT classifier.

* Collecting datasets of available APTs from different sources and publishing them for the use of the cybersecurity community.
The rest of this paper is structured as follows. Section 2 contains related works on malware classification. In section 3, we explain our two methods in detail. Section 4 presents our experimental results about the proposed methods. In this section, we introduce the collected dataset and the results. At the end of this section, we analyze the results and compare the advantages of each method. The last section contains our concluding remarks and future works.

## II. RELATED WORKS

APT attacks may take several steps to reach the target (e.g., reconnaissance, breach, infiltration, ex-filtration, and stealth persistence) [1]. Attackers utilize a variety of tools, techniques, and malwares in each step and may exploit known or unknown vulnerabilities, and communicate by encrypted protocols to achieve their final goal. Since the attack may have left a sign on one of the steps that make it detectable, researchers proposed single or multi-step detection methods [2], [3]. Also, some studies just analyzed a specific APT attack to know more about the attack patterns and the kill chain attackers have used in the attack process [4], [5].

Generally, we can classify the APT detection techniques into the anomaly-based detection and pattern matching detection. While in anomaly-based detection, the defender tries to learn the normal behavior of the users, systems, and their communications, so to detect any behavior which deviates from it [6], in the pattern-based detection, the defenders describe the known attack pattern as a set of signatures [7]. However, due to the complexity and unknown nature of the APT attacks, the use of the pattern-matching detection approach is not very effective.

Researchers have used a variety of sources to extract attack features such as e-mails, malware, DNS logs, system logs, and network traffic. Each of these sources can be useful in identifying a step or steps in an ongoing attack. Numerous studies have discussed malware detection, but due to the differences in the nature of the malware used in APT attacks, a different approach is required to address the challenges in this area. Here we review some APT malware detection research.

Researchers in [8] used deep neural networks for dynamic malware analysis with acceptable accuracy. The input of their method is a set of the sparse binary vector of length 179000. The authors concluded that adding the second and third hidden layers to the deep neural network does not necessarily improve the results.

Researchers in [9] developed a method called MTNET, a binary classifier based on a deep neural network that uses static and dynamic characteristics of system calls for detection. In this method, malware is classified into one of 100 known families.

Researchers at [10] used the Cuckoo Sandbox analysis report to extract high-frequency words. They used a deep neural network to extract the signature of the attacks. They used the SVM algorithm to classify the malwares.

Researchers in [11] used a system called CHATTER that assigns a letter to each system event. Then, it puts these letters

together and executes the created sequence. CHATTER also used the created N-grams to obtain a suitable sequence. The researchers extracted characteristics of memory, registry, or network interface events.

Researchers in [12] extracted system calls and used machine learning methods for classification. Some other researchers used dynamic analysis to extract system calls, which is not sufficient for feature extraction because some malware detects the dynamic analysis environment and behaves differently. For this reason, a combination of static and dynamic analysis is required to extract system calls. The Cuckoo sandbox analyzes events in this way.

Researchers in [13] used commands such as JMP, LOOP, and CALL to create n-grams and extract only one n-gram from each block of code. In this research, an expert is used to determine the words to find words in the text of the code, which is the disadvantage of this work.

## III. PROPOSED METHODS

In this section, we will introduce two proposed methods using Deep Neural Networks and Machine Learning algorithms. Both of these methods use the TF-IDF measure in the process of feature selection and input vector representation. Therefore, we will review this measure. Then, we will discuss the proposed methods in the following two subsections.

### A. TF-IDF measure

The TF-IDF measure introduced by the researchers in the field of Information Retrieval [14]. The main objective of this measure is to categorize documents by their most important words. It is composed of two parts, TF (Term Frequency) and IDF (Inverse Document Frequency), and identifies the importance of each word. So, this approach uses the set of most important words of a document for categorizing it.

Suppose we have a collection of N documents. The frequency of term i in document j is $n(i, j)$ normalized by dividing it by the maximum number of occurrences of any term in the document j as (1):

$$TF_{ij} = \frac{n(i, j)}{\sum_{k} n(k, j)} \qquad (1)$$

Since the term with high frequency is not an appropriate discriminator, the IDF measure describes whether a term i is rare in all documents. So the $log_2$ of total number of documents (N) over the documents contain the term i, ($n_i$) defines the IDF$_i$ as (2):

$$IDF_i = log_2 \frac{N}{n_i} \qquad (2)$$

So the discrimination is as important as term frequency which is defined in the TF-IDF equation as (3):

$$TF - IDF_{ij} = TF_{ij} \times IDF_{i} \qquad (3)$$
## B. Method 1: Hybrid Malware Classifier based on DNN

There are two main methods of malware behavior analysis: static analysis and dynamic analysis. The purpose of static analysis is to identify malware's behavior without execution, and dynamic analysis obtains it by executing the malware. Each method provides valuable information for the detection algorithm. The hybrid analysis is based on two analysis methods together to detect malwares.

The first method we have proposed in this section combines the hybrid analysis and Deep Neural Networks (DNN) to classify an APT malware and identify its family label. This method contains five main steps shown in Fig. 1. In the first step, each malware is passed to the Cuckoo sandbox, and Cuckoo generates the static and dynamic analysis report for the malware. Next, the terms with the best TF-IDF score are collected. A collection of terms with high TF-IDF creates a vector of features and would pass to the DNN learning model. In the last step, transfer learning is used for target identification. In the following subsections, each step is described in detail.

<figure><img src="image_2.png" /><figcaption>Fig. 1: The main steps of Method 1: Hybrid Malware Classifier based on DNN</figcaption></figure>

### 1) Use Cuckoo to get static and dynamic malware analysis logs:

The hybrid analysis of the malware binaries contains the static and dynamic analysis of their codes. We have used Cuckoo sandbox. It provides a set of data in a JSON format for each sample file called a report. Each report contains various information about the static features extracted from a file, such as strings, and the dynamic behavior of the file at run time, such as API calls. An example part of Cuckoo's report is shown in Fig. 2. So, in this step, we get the malware binaries as the input and the list of generated reports of the hybrid analysis of each sample separately as the output.

### 2) Select the terms with the best TF-IDF:

Since each report is a text file, we considered it a document and used a TF-IDF measure to find the valuable terms. For this purpose, first, we extracted all terms of each report to a list. As an example, the line `"api: CreateFileW"` results in two terms: `"api"` and `"CreateFileW"`. Since the generated report contains irrelevant terms related to sandbox settings (e.g., `"VM-Win7"`), we eliminated them from the list. This elimination makes the process faster, and the list includes only terms related to the malware code and execution. Once the list

```json
{
  "markcount": 1,
  "families": [],
  "description": "The executable uses a known packer",
  "severity": 1,
  "marks": [
    {
      "category": "packer",
      "ioc": "Armdillo v1.71",
      "type": "ioc",
      "description": null
    },
    "references": [],
    "name": "peid_packer"
  },
  {
    "markcount": 1,
    "families": [],
    "description": "Foreign language identified in PE resource",
    "severity": 2,
    "marks": [
      {
        "name": "RT_VERSION",
        "language": "LANG_CHINESE",
        "offset": "0x00013060",
        "filetype": "data",
        "sublanguage": "SUBLANG_CHINESESIMPLIFIED",
        "type": "generic",
        "size": "0x00000328"
      },
      "references": [],
      "name": "origin langid"
    }
  ]
}
```

<caption>Fig. 2: Example part of Cuckoo's report</caption>

is ready, we calculate the TF-IDF of each term in the list. So, in this step, we get the reports as input and generate a list of all relevant terms. Then, we generate a dictionary as an output that contains the terms and their TF-IDF ratio as a key-value pair.

### 3) Create a representation vector as the input of DNN:

In this step, first, we collect the 5000 terms with high TF-IDF ratios from the dictionary we receive from the previous step. These terms make a vector of valuable terms for all reports. Next, we consider a binary vector $V$ for each report with the same length. We go through the report and search it for all 5000 collected terms. If the term $i$ exists in the malware report, we set the $V(i) = 1$, and if it does not appear in it, we set the $V(i) = 0$. These binary vectors produced per malware report are the output of this step.

### 4) Train the DNN to create a detection classifier model:

The collected vectors of the previous step are the inputs of the deep neural network (DNN) to detect a correct malware family label. The DNN is a multi-level perceptron with a denoising auto-encoder for each layer and contains five layers: the input layer, three hidden layers, and the output layer (5000 – 2000 – 1000 – 250 – 7). The last layer has seven neurons which are our seven malware family labels. Fig. 3 shows the structure of DNN.

An auto-encoder is an unsupervised deep learning structure that sets the output values equal to the input. It has a hidden layer that has fewer neuron numbers. This bottleneck forces the network to learn a higher-level representation of the input. So the input is mapped to the hidden layer, and the output layer tries to reconstruct the input. This structure has two types of base autoencoders and denoising autoencoders. The difference is that the denoising autoencoder adds noise and corrupts the
<figure><img src="image_3.png" /><figcaption>Fig. 3: The structure of proposed DNN</figcaption></figure>

input. Then tries to create the uncorrupted version of inputs. This addition makes the network resistant to the overfitting problem. For each auto-encoder, the size of the input and output layer is the size of the current layer, and the size of the hidden layer is the size of the next layer in DNN. Fig. 4 shows the structure of auto-encoders of layers 1 and 2. Since we used the autoencoder in each layer, we trained them separately. So, after the training of each layer, its weights are frozen. Then, we move to the next layer to train its weights.

The last layer of this DNN structure is softmax layer. The softmax layer converts the output values to probabilities. For example, if we have four labels and the output has the values [4, 1, 2, 1], the softmax layer returns [0.5, 0.2, 0.4, 0.2], the probabilities of a sample belonging to each family.

<figure><img src="image_4.png" /><figcaption>Fig. 4: The structure of auto-encoders of layer 1 and 2</figcaption></figure>

### 5) Transfer Learning for target identification:

Since APT attacks are targeted attacks, it is necessary to identify the target. We use the "Transfer Learning" approach to achieve this goal. If the results of transfer learning are good, the features extracted by the network can explain the behavior of the attack, and we can use them to identify other features of the attack.

To address this issue, we added another layer to the DNN structure with four neurons that are the targets' labels. With the exception of the last layer, which contains the labels, other layers, and their weights are frozen. The target of an attack shows the targeted sector. The labels are "IndustrialSystems", "FinancialSystems", "ITSystems" and "Others". In the

training phase, we just trained the last added layer. Therefore, it is shorter than the previous network.

### C. Method 2: n-gram of API Calls and Machine Learning Classifier

In the second method, we use the n-gram of API calls obtained from Cuckoo sandbox analysis. The difference between this method and the previous one is that we use only the API calls. We calculate the best n-grams with TF-IDF ratio for each malware family (i.e., feature vectors). Then, we used four machine learning algorithms to predict each sample's label. Fig. 5 showed the main steps of this method.

<figure><img src="image_5.png" /><figcaption>Fig. 5: The main steps of Method 2: n-gram of API calls and Machine Learning Classifier</figcaption></figure>

1) Use Cuckoo to get static and dynamic malware analysis reports:

Like the first method, we use the Cuckoo sandbox to analyze the malware by static and dynamic methods and use its JSON report to extract the behavior of each sample.

2) Extract the list of API calls:

We extract the list of API calls obtained from the previous step. Since we need to identify the family label of each malware, we collect API calls of malwares of the same family in one list. Thus we have a list of API calls per family label in the training dataset as an output of this step.

3) Select n-grams of API calls with the best TF-IDF ratio:

This step consists of multiple tasks. First, we extracted n-grams of API calls from each malware family list obtained from the previous step. Then, we repeated the following two sub-steps for $n = 2, 3, 4, 5$ in this step:

* The whole sequence of API calls of all members of a malware family is considered a document. Thus, each n-gram of this sequence is a term. We calculate the TF-IDF value of all sequences.

* Select *d* best n-grams from the previous sub-step. The value of *d* is important since it has an effect on the accuracy of each machine-learning algorithm. We call *d* as the dimension, and it is different for each algorithm.
### 4) Create the feature vectors:

After the selection of n-grams with the best TF-IDF, we create the feature vector by calculating the TF value of each n-gram in each sequence which is a constant number of n-grams. For example, suppose that we have a sequence of 3-grams with a length of 10 as follows:

((A,B,B), (B,B,C), (B,C,D), (C,D,B), (D,B,E), (B,E,F), (E,F,B), (F,B,B), (B,B,C), (B,C,D))

if (F,B,B), (B,C,D), and (B,E,F) are the best n-grams, the TF value would be (0.2, 0.2, 0.1).

By repeating this step for each sequence of n-grams, we would have a set of feature vectors as the inputs of machine learning algorithms in the next step.

### 5) Use Machine Learning algorithms to predict the labels:

In this method, we used four machine learning algorithms to classify the malware: SVM, Random Forest, K-Nearest Neighbors, and Naive Bayes.

## IV. EXPERIMENTAL RESULTS

In this section, we introduce our dataset, and then we report the results from the implementation of each method. At the end of this section we compare the results.

### A. Dataset

Our collected dataset has 814 samples of APT malwares. The process is as follows:

* Extract the MD5 hash of APT malware for each family from Kaspersky reports [15].

* Search through the communities such as VirusTotal [16] and VirusShare [17] to find and collect the samples.

* Validate the label of each sample by Kasprskey's documents.

The collected data set consists of seven APT malware families categorized according to their behavior. These malware families are as follows.

* **Sofacy or APT28:** This family includes Fancy Bear Crew and Tsar Crew. These attacks use zero-day vulnerabilities and spear-phishing to breach the system. This threat actor has been active since mid-2000 and targets mainly the political sector.

* **Equation:** This family of attacks targeted industries and energy infrastructures. Some of the samples were undetected for about ten years. The most important part is the one that can reprogram the hard drive firmware. The Stuxnet and Flame are the most dangerous attacks in this family.

* **Winnti:** This is a Remote Administration Malware which first observed in 2012. The targets were government and military/aerospace systems.

* **Black Energy:** This malware targeted industrial systems in Ukraine. It is designed based on Microsoft Word and Microsoft Excel Macros. It uses an LNK file and can copy it to startup to execute during each boot process. Also, this malware was used for DDoS attacks.

* **Turla:** This malware is known as Snake and Uroburos and is a complex cyber-espionage tool. This malware uses a Watering Hole attack and exploits vulnerabilities in Adobe Flash and Internet Explorer. After the infection, it installs a backdoor on the victim system and uses it for further communication with the server.

* **CozyDuke:** This malware was designed for cyber-espionage goals and targeted military and government systems. The samples were first observed in 2014 and contain CosmicDuke and MiniDuke attacks.

* **Carbank:** This malware has targeted financial systems since 2013. It uses Microsoft Word's vulnerability and provides remote access to infected systems. This malware can capture videos from common and administrator users.

The total number of samples in each family and the number of samples used for test and train steps are shown in Tab. I.

<table><caption>TABLE I: Number of samples for each family</caption><thead><tr><th></th><th># of samples</th><th># of train samples</th><th># of test samples</th></tr></thead><tbody><tr><td>Sofacy</td><td>148</td><td>111</td><td>37</td></tr><tr><td>Equation</td><td>110</td><td>82</td><td>28</td></tr><tr><td>Winnt</td><td>112</td><td>84</td><td>28</td></tr><tr><td>Black Energy</td><td>89</td><td>67</td><td>22</td></tr><tr><td>Turla</td><td>131</td><td>98</td><td>33</td></tr><tr><td>CozyDuke</td><td>126</td><td>95</td><td>31</td></tr><tr><td>Carbanak</td><td>98</td><td>73</td><td>25</td></tr><tr><td>Total</td><td>814</td><td>610</td><td>204</td></tr></tbody></table>

### B. Results of Method 1

We used ReLU and PReLU as the activation functions. For the negative inputs, ReLU uses $0.01x$ while PReLU uses $\alpha x$ and $\alpha$ trained in the training phase. We achieved a 10% improvement using the PReLU as the activation function. The results are shown in Tab. II.

<table><caption>TABLE II: Results of the Method 1 with %85 threshold</caption><thead><tr><th></th><th># of samples</th><th>ReLU</th><th>PReLU</th></tr></thead><tbody><tr><td>Sofacy</td><td>37</td><td>81.08</td><td>86.49</td></tr><tr><td>Equation</td><td>28</td><td>78.57</td><td>85.71</td></tr><tr><td>Winnt</td><td>28</td><td>82.14</td><td>89.29</td></tr><tr><td>Black Energy</td><td>22</td><td>81.82</td><td>90.91</td></tr><tr><td>Turla</td><td>33</td><td>84-85</td><td>93.94</td></tr><tr><td>CozyDuke</td><td>31</td><td>87.1</td><td>93.55</td></tr><tr><td>Carbanak</td><td>25</td><td>84</td><td>92</td></tr><tr><td>Total</td><td>204</td><td>82.84</td><td>90.19</td></tr></tbody></table>

To test the target identification, we delete the last layer and add a layer with four labels. Then, we train the new layer for transfer learning. The results are shown in Tab. III. The results show that the features extracted from malware's behavior are good enough to 10% difference in accuracy. We believe we can gain better results from transfer learning using a larger dataset.

To ensure the correct number of hidden layers designed in this method, we have compared its accuracy with the number of hidden layers (HL) 2 and 4. Tab. IV shows that shallower
<table><caption>TABLE III: Results of Method 1 for target identification</caption><thead><tr><th></th><th># of samples</th><th>True Positive</th><th>Accuracy(%)</th></tr></thead><tbody><tr><td>Sofacy</td><td>37</td><td>31</td><td>86.49</td></tr><tr><td>Equation</td><td>28</td><td>22</td><td>78.57</td></tr><tr><td>Winnt</td><td>28</td><td>23</td><td>82.14</td></tr><tr><td>Black Energy</td><td>22</td><td>17</td><td>77.27</td></tr><tr><td>Turla</td><td>33</td><td>27</td><td>81.82</td></tr><tr><td>CozyDuke</td><td>31</td><td>26</td><td>83.87</td></tr><tr><td>Carbanak</td><td>25</td><td>20</td><td>80</td></tr><tr><td>Total</td><td>204</td><td>166</td><td>81.37</td></tr></tbody></table>

and deeper DNNs have lower accuracy, and the best number of hidden layers is three in this method.

<table><caption>TABLE IV: Results of Method 1 with two hidden layers</caption><thead><tr><th></th><th># of samples</th><th>HL = 2</th><th>HL = 4</th></tr></thead><tbody><tr><td>Sofacy</td><td>37</td><td>81.08</td><td>83.78</td></tr><tr><td>Equation</td><td>28</td><td>82.14</td><td>82.14</td></tr><tr><td>Winnt</td><td>28</td><td>85.71</td><td>78.57</td></tr><tr><td>Black Energy</td><td>22</td><td>76.36</td><td>77.27</td></tr><tr><td>Turla</td><td>33</td><td>87.87</td><td>81.82</td></tr><tr><td>CozyDuke</td><td>31</td><td>87.09</td><td>90.32</td></tr><tr><td>Carbanak</td><td>25</td><td>84</td><td>80</td></tr><tr><td>Total</td><td>204</td><td>84.80</td><td>82.35</td></tr></tbody></table>

### C. Results of Method 2

In this section, we presented the best results gained by the best dimension and best N for N-gram for each algorithm.

1) **SVM**: The first algorithm we used for the classification is SVM. Since the complexity of the problem is high, we used the RBF kernel. SVM is the only algorithm that can identify the family of some Flame attack samples. The best parameters for the SVM algorithm as illustrated in Fig. 6 are dimension = 47 and 4-gram.

2) **RF**: The second algorithm we used is Random Forest. In this algorithm, the number of base learners is the number of decision trees in the Random Forest (i.e., 120). The maximum depth is the number of nodes from root to leaf. The number of features to consider for each node is $log_2(number\ of\ all\ features)$. This algorithm also has the disadvantage of misclassifying the Flame's samples. The best parameters for the RF algorithm as illustrated in Fig. 7 are dimension = 45 and 3-gram.

3) **K-NN**: The K Nearest Neighbor algorithm has lower accuracy than Random Forest and SVM because of its simplicity. The best parameters for the K-NN algorithm are $k = 5$, dimension = 33, and 3-gram (as illustrated in Fig. 8). This algorithm works better with short dimensions and a short sequence of API calls.

4) **NB**: The last algorithm is Naïve Bayes which has the worst results in this research. The lower accuracy is the result of considering features independently. The Gaussian distribution is used for this algorithm. This algorithm has better results with longer dimensions and API call sequences and also worked better with 4-gram (as illustrated in Fig. 9).

Tab. V shows the best accuracy results of each classifier algorithm.

<table><caption>TABLE V: Accuracy results of different algorithms in percentage</caption><thead><tr><th></th><th># of samples</th><th>SVM</th><th>RF</th><th>K-NN</th><th>NB</th></tr></thead><tbody><tr><td>Sofacy</td><td>37</td><td>94.59</td><td>94.59</td><td>89.19</td><td>72.97</td></tr><tr><td>Equation</td><td>28</td><td>92.86</td><td>85.71</td><td>78.57</td><td>71.43</td></tr><tr><td>Winnt</td><td>28</td><td>89.29</td><td>96.42</td><td>85.71</td><td>68.86</td></tr><tr><td>Black Energy</td><td>22</td><td>90.90</td><td>95.45</td><td>81.81</td><td>68.18</td></tr><tr><td>Turla</td><td>33</td><td>90.90</td><td>93.93</td><td>81.81</td><td>72.73</td></tr><tr><td>CozyDuke</td><td>31</td><td>90.32</td><td>93.54</td><td>83.87</td><td>74.19</td></tr><tr><td>Carbanak</td><td>25</td><td>92</td><td>96</td><td>84</td><td>68</td></tr><tr><td>Total</td><td>204</td><td>91</td><td>93.63</td><td>83.33</td><td>71.08</td></tr></tbody></table>

<figure><img src="image_6.png" /><figcaption>Fig. 6: The accuracy of different dimensions vs n-gram for SVM</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 7: The accuracy of different dimensions vs n-gram for RF</figcaption></figure>

### D. The comparison of the results

In this section, we compare our two proposed methods and discuss their benefits and limitation of them. We compare these two methods against each other based on four criteria. The results are shown in Tab. VI.

The first method based on DNN needs a larger dataset because of the extraction of high-level features from low-level features. So it is more dependent on the size of the
<figure><img src="image_8.png" /><figcaption>Fig. 8: The accuracy of different dimensions vs n-gram for K-NN</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 9: The accuracy of different dimensions vs n-gram for NB</figcaption></figure>

dataset. But it has the advantage of significant improvement by increasing the dataset size. In our experiments, we witnessed a significant improvement in accuracy when the number of samples increased from 600 to 814.

The second proposed method is based on the sequence of API calls, while the first method works with hybrid features obtained by static and dynamic analysis. However, the second method has better results.

The first method needs low pre-processing of reports, and we only find the terms with the best TF-IDFs. In the second method, we should extract the N-grams and frequency. The DNN needs less pre-processing because of automatic high-value feature extraction.

The first method (DNN) has more parameters to improve the results. The number of layers for DNN and their activation functions, type of the DNN, using denoising autoencoders, number of neurons in each layer, and some other parameters are used to improve the accuracy of the classifier. But for each algorithm used for the N-gram of API calls, we have a low number of parameters to get better results.

<table><caption>TABLE VI: Comparison of two proposed methods</caption><thead><tr><th></th><th>Method 1</th><th>Method 2</th></tr></thead><tbody><tr><td>Dependency on dataset size</td><td>high</td><td>low</td></tr><tr><td>Feature used</td><td>static and dynamic analysis</td><td>API calls</td></tr><tr><td>Cost of pre-Process</td><td>low</td><td>high</td></tr><tr><td>Dependent parameters</td><td>high</td><td>low</td></tr></tbody></table>

## V. CONCLUSION AND FUTURE WORKS

Our motivation for doing this research was to identify the family of APT malwares and their upcoming versions. In this paper, we presented two methods based on Deep Neural Network and N-gram of API calls for Advanced Persistent Threats family classification. The first method uses raw features from sandbox reports of static and dynamic analysis and extracts high-value features automatically. This method significantly improves the results when applied to a larger dataset and needs less pre-processing for creating vectors. However, it has more dependency on the size of the training dataset. Another advantage of this method is the possibility of transfer learning by editing the last layer labels and using the DNN without a new training phase to identify the target of attacks.

The second method used an n-gram of API calls obtained from malware analysis to create input vectors with the best TF-IDF. In this method, we used four different machine learning algorithms. The best results are for Random Forest and the SVM. The SVM is the only algorithm capable of classifying the Flame attack samples. This attack has a bigger malware size than the others.

We collected a dataset of APT malwares from different sources, and we believe that by collecting a bigger dataset the result of the DNN-based method will be better than the other methods. However, finding the APT samples is challenging because of the high classification level of these attacks by many targeted organizations.

There are some directions to improve this research for the future as follows.

* First, we can collect the input vectors of the second method with other features like opcodes to create valuable feature vectors and use them in the DNN method to classify the APTs.

* We can improve both methods to detect the APTs for a special type of target systems. It can help us to provide a more accurate classifier for special types of attack in contrast to providing a general model for all malwares.

* The other valuable research can be the simulation of APTs with the existing scenarios in different reports. By extracting the different features of APTs for each attack, it can be possible to simulate the APT attacks and create a dataset to solve this problem and use it for future research in this field.

## REFERENCES

[1] S. Singh, P. K. Sharma, S. Y. Moon, D. Moon, and J. H. Park, "A comprehensive study on apt attacks and countermeasures for future networks and communications: challenges and solutions," *The Journal of Supercomputing*, vol. 75, no. 8, pp. 4543-4574, 2019.
[2] G. Zhao, K. Xu, L. Xu, and B. Wu, "Detecting apt malware infections based on malicious dns and traffic analysis," *IEEE access*, vol. 3, pp. 1132–1142, 2015.

[3] I. Friedberg, F. Skopik, G. Settanni, and R. Fiedler, "Combating advanced persistent threats: From network event correlation to incident detection," *Computers & Security*, vol. 48, pp. 35–57, 2015.

[4] A. L. Johnson, "Cybersecurity for financial institutions: The integral role of information sharing in cyber attack mitigation," *NC Banking Inst.*, vol. 20, p. 277, 2016.

[5] N. Falliere, L. O. Murchu, and E. Chien, "W32. stuxnet dossier," *White paper, Symantec Corp., Security Response*, vol. 5, no. 6, p. 29, 2011.

[6] P. Garcia-Teodoro, J. Diaz-Verdejo, G. Maciá-Fernández, and E. Vázquez, "Anomaly-based network intrusion detection: Techniques, systems and challenges," *computers & security*, vol. 28, no. 1-2, pp. 18–28, 2009.

[7] X. Yan and J. Y. Zhang, "Early detection of cyber security threats using structured behavior modeling," *ACM Transactions on Information and System Security*, vol. 5, 2013.

[8] G. E. Dahl, J. W. Stokes, L. Deng, and D. Yu, "Large-scale malware classification using random projections and neural networks," in *2013 IEEE International Conference on Acoustics, Speech and Signal Processing*. IEEE, 2013, pp. 3422–3426.

[9] W. Huang and J. W. Stokes, "Mtnet: a multi-task neural network for dynamic malware classification," in *International conference on detection of intrusions and malware, and vulnerability assessment*. Springer, 2016, pp. 399–418.

[10] O. E. David and N. S. Netanyahu, "Deepsign: Deep learning for automatic malware signature generation and classification," in *2015 International Joint Conference on Neural Networks (IJCNN)*. IEEE, 2015, pp. 1–8.

[11] A. Mohaisen, A. G. West, A. Mankin, and O. Alrawi, "Chatter: Classifying malware families using system event ordering," in *2014 IEEE Conference on Communications and Network Security*. IEEE, 2014, pp. 283–291.

[12] S. S. Hansen, T. M. T. Larsen, M. Stevanovic, and J. M. Pedersen, "An approach for detection and family classification of malware based on behavioral analysis," in *2016 International conference on computing, networking and communications (ICNC)*. IEEE, 2016, pp. 1–5.

[13] M. Hassen, M. M. Carvalho, and P. K. Chan, "Malware classification using static analysis based features," in *2017 IEEE Symposium Series on Computational Intelligence (SSCI)*. IEEE, 2017, pp. 1–7.

[14] H. Zhang, X. Xiao, F. Mercaldo, S. Ni, F. Martinelli, and A. K. Sangaiah, "Classification of ransomware families with machine learning based on n-gram of opcodes," *Future Generation Computer Systems*, vol. 90, pp. 211–221, 2019.

[15] "https://apt.securelist.com/."

[16] "https://www.virustotal.com/gui/."

[17] "https://virusshare.com/."