# An Empirical Evaluation of Supervised Learning Methods for Network Malware Identification Based on Feature Selection

Authors: C. Manzano; C. Meneses; P. Leger; H. Fukuda
Journal: Complexity
 (2022)

---

## Abstract

Malware is a sophisticated, malicious, and sometimes unidentifiable application on the network. The classifying network traffic method using machine learning shows to perform well in detecting malware. In the literature, it is reported that this good performance can depend on a reduced set of network features. This study presents an empirical evaluation of two statistical methods of reduction and selection of features in an Android network traffic dataset using six supervised algorithms: Naïve Bayes, support vector machine, multilayer perceptron neural network, decision tree, random forest, and K-nearest neighbors. The principal component analysis (PCA) and logistic regression (LR) methods withp value were applied to select the most representative features related to the time properties of flows and features of bidirectional packets. The selected features were used to train the algorithms using binary and multiclass classification. For performance evaluation and comparison metrics, precision, recall, F-measure, accuracy, and area under the curve (AUC-ROC) were used. The empirical results show that random forest obtains an average accuracy of 96% and an AUC-ROC of 0.98 in binary classification. For the case of multiclass classification, again random forest achieves an average accuracy of 87% and an AUC-ROC over 95%, exhibiting better performance than the other machine learning algorithms. In both experiments, the 13 most representative features of a mixed set of flow time properties and bidirectional network packets selected by LR were used. In the case of the other five classifiers, their results in terms of precision, recall, and accuracy, are competitive with those obtained in related works, which used a greater number of input features. Therefore, it is empirically evidenced that the proposed method for the selection of features, based on statistical techniques of reduction and extraction of attributes, allows improving the identification performance of malware traffic, discriminating it from the benign traffic of Android applications.


---

## Body

## 1. Introduction

Malware is short for malicious software, it is a generic term widely used to name all the different types of unwanted software programs [1]. There are various types of malware such as viruses, scareware, ransomware, ad-ware, spyware, smsware, etc. [2]. Cybercriminals have used malware as a network attack weapon to encrypt and hijack personal computer data, steal confidential information from information systems, penetrate networks, bring down servers, and cripple critical infrastructure [2]. These attacks often cause serious damage and generate significant economic losses [3].According to a June 2020 report delivered by Kaspersky Lab, the number of malware attacks from 2018 to 2019 increased by 37% and reached 1,169,153 new cases at the end of last year. Also, McAfee Labs observed that during the first quarter of 2020 the number of malware threats to mobile applications was 375 per minute [4]. Today, one of the mobile platforms most affected by malware attacks is Android [5]. Generating new solutions that allow the detection and identification of new types of malware is a challenge that cybersecurity research communities must address to prevent the exploitation and misuse of current systems.In the literature, to support the detection and identification of malware, three analysis techniques are proposed: static analysis, dynamic analysis and network analysis [6]. Static analysis is mainly based on the study of malware source codes and is easily bypassed through code obfuscation [7]. Dynamic analysis focuses on using operating system calls to extract reliable information from malware execution traces [8]. The main disadvantage of dynamic analysis is to find the exact traceability of the behavior of the malware by being in a controlled environment called sandbox [9].Unlike static and dynamic analysis techniques, which are based on the recognition of malware code and behavior within a host [10], network analysis allows the recognition of malware behavior according to the direct or passive features of conversations of a network flow [6]. The flow of the network can be seen as a set of conversations that is represented as a statistical summary of the network traffic between a source IP (Internet Protocol) and destination IP [11]. Network analysis has raised additional challenges such as data encryption and port obfuscation in network malware behavior [12]. One of the network analysis techniques to identify malware is the classification of network traffic with machine learning [13]. In the empirical works of [14–16], the network traffic classification method with machine learning has shown good results in the identification of malware. However, a common problem with this method is to adapt, on certain occasions, to high-dimensional datasets with irrelevant and redundant features to accurately classify and identify the types of malware ([17, 18]).Feature reduction is a critical activity within the data preprocessing stage of a machine learning project [19] and especially for a network traffic classification problem due to the emergence of new network service traffic patterns and to the great demand for bandwidth [20]. The goal of feature reduction is to obtain a reduced representation of the original dataset that has not been processed. Wavelet Transform, PCA (Principal Component Analysis), Clustering, sampling, and traditional feature selection techniques such as Wrapper, Embedded and Filter, are methods used within the feature reduction phase at the stage of data preprocessing of a machine learning project [21]. Reducing or selecting a minimum number of features to represent the behavior of network traffic is a key task to achieve good performance in the malware detection and identification process [22].Recently, in [21, 23] it is evidenced that researchers have adopted statistical methods of reduction and selection of features in order to improve the performance of detection and identification of malware. This is the case of [24], where the PCA statistical method is applied to reduce the features of the Application Programming Interface (API) type of the MamaDroid malware detection system. The authors in [24] initially worked with 116,281 features and managed to reduce their dimension to 10 main components. MamaDroid scored a good 99.9% performance for F-measure, and averaged over 90% accuracy and recall for all its malware detection experiments. In [25], experimental work was performed using the support vector machine (SVM) classifier to detect malware. In [25] 20 features of OpCode (operation code) were used and it was possible to reduce the initial dimension to 8 components by means of the PCA statistical method. The PCA method, applied in [25], managed to represent 99.5% of the total variance of its components. In [25], K-nearest neighbors (KNN) performed well at 83.41% accuracy, and 4.2% false negatives (FN) in detecting malware. Another study carried out in [26], applied the Sparse Logistic Regression (SLR) method to discriminate the less significant features of the model and improve the classification of malware attacks with its intrusion detection system (IDS). The SLR method was able to discriminate 4 features from the initial dataset of 20 features. In [26] a p-value of 0.5 was used, and overfitting and feature redundancy were controlled by simultaneously selecting and classifying them. SPLR achieved good malware detection performance of 97.6% overall accuracy and a total of 0.34% false positives (FP). In [27], a method called 4-LFE (L1-L2-LR-LDA Feature Extraction) is presented, composed of statistical techniques such as L1-L2 Penalty, logistic regression and linear discriminant analysis (LDA), to reduce the dimension of features and detect malware. The experimental results of the 4-LFE method show that it managed to classify the malware with 99.99% accuracy.This paper presents the results of an empirical comparison of the performance of shallow learning algorithms, such as Naïve Bayes, support vector machine, multilayer perceptron neural network, decision tree, random forest, and K-nearest neighbors, to identify malware traffic. Two statistical techniques, PCA and logistic regression withp-value, were considered to reduce and select the most significant features related to the time of flows and bidirectional packets of the dataset with CICAndMal2017 Android network traffic. This work seeks to contribute through:(1)
The proposed feature selection methodology based on a combination of statistical and computational methods.(2)
The comparative analysis of different machine learning algorithms when applied to the identification of malware traffic based on different sets of preselected features. This provides empirical evidence that a feature selection method based on statistical and computational techniques generates better predictive results in relation to the use of all features without prior selection, particularly in the domain of identifying malware versus benign traffic.The rest of the work is structured as follows. The following section discusses material and methods used in this study. Then, this paper describes the dataset and the methods used to perform the feature selection. After, we explain the four-phase methodology proposed for this work. Using this methodology, this paper presents performance of the experiments with the associated results. Then, the results are compared to those obtained in related work, regarding the identification of malware in network traffic considering methods of reduction and selection of features. Finally, the conclusions and future work are presented.


## 2. Methods and Materials

In this section, the CICAndMal2017 dataset used is first described, and second, the data preprocessing for binary and multiclass classification is explained. Finally, the feature selection methods used as part of the methodology proposed in this work are presented.


### 2.1. Dataset

The CICAndMal2017 dataset is made up of a combination of more than 80 flow time and network packet features to detect and identify malware traffic alongside benign Android applications. This set was built by Lashkari et al. [28] of the Canadian Cybersecurity Institute (CIC). CICAndMal2017 offers 2,126 files in CSV (comma separated values) format and more than 20 gigabytes in PCAP (packet capture) files with malware traffic conversations and benign Android mobile network applications, captured in the years 2015–2017. Network traffic of benign applications are labeled “benign.” Malware traffic is labeled into four categories: adware, ransomware, scareware, and smsware. Each category of malware consists of different families as presented in Table 1 [11]. Originally both sets of CSV and PCAP files are structured with more than 80 Android network traffic features.Table 1
Category and family of malware.
CategoryFamily typeAdwareEdwinKoodousKemogeDowginMobidashYoumiFeiwoSelfmiteRansomwareR ShuanetGooliganChargerPletorLockerPinJisutRansomBOSvpengPornDroidKolerScarewareWannaLockerSimplockerAndroidDefenderFakeAVFakeApp.ALFakeJobOffAVforAndroidPenethoVirusShieldFakeAppSmswareFakeTaoBaoAVpassAndroidSpy.277BeanBotJifakeFakeNotifyBiigeNandroboxFakeInstMazarbotPlanktonFakeMartSMSsniffer


### 2.2. Feature Selection Methods

Two statistical methods of feature selection are described below, known as PCA and Logistic Regression, which were selected and used for their good performance in features selection work in the detection and identification of malware [26, 27, 29–33]. The PCA and logistic regression methods were used to select the most representative features of network traffic from the input data of the CICAndMal2017 dataset.


### 2.3. Feature Selection Based on Principal Component Analysis (PCA)

PCA is a method used to reduce the dimensionality of a large dataset to a smaller one, containing a large part of the information from the original set [24]. Reducing the number of features in a dataset sometimes means losing valuable information, but it also means simplifying the problem, since it is easier to explore and visualize data in small sets [34]. The PCA method therefore allows condensing the information provided by multiple variables into only a few components and having the value of the original variables to calculate these components [35]. Therefore, PCA decomposes a dataset into eigenvectors and eigenvalues. An eigenvector is a direction, for example (x, y), and an eigenvalue is a number that represents the value of the variance in that direction [34]. The main component will be the eigenvector with the highest eigenvalue. There are as many eigenvector/eigenvalue pairs in a dataset as there are dimensions. The eigenvectors do not modify the data, but rather allow us to see them from a different point of view, more related to the internal structure of the data, and with a much more intuitive view of them [30]. Once the eigenvalues, which are a measure of the variance of the data, have been ordered, it is necessary to decide which is the smallest number of eigenvectors or principal components to maintain. To do this, a metric known as explained variance is used, which shows how much variance can be attributed to each of these principal components. Furthermore, as defined in [35], the principal components can be conceptualized as new axes that offer a new coordinate system to evaluate the data, making the differences between the observations in the dataset more visible. The PCA tries to put as much information as possible in the first component, then as much information as possible in the second component, and so on. This process is done until you have a total of principal components equal to the original number of features. As mentioned in [35], there is no single answer or method that allows identifying the optimal number of principal components to select. A very widespread way of proceeding consists of evaluating the proportion of accumulated explained variance and selecting the minimum number of components beyond which the increase is no longer substantial.In other words, PCA corresponds to a linear transformation that takes the input data to a new space of orthogonal axes. In this new space, the axes are ordered such that the first axis captures the largest variance of the original data (called the first principal component), and the last axis captures the smallest variance. Formally [36], let X be a data matrix of dimensions n×p, where each column of data is previously normalized to have zero mean. Here n and p correspond to the number of observations and the number of columns or features of the data set, respectively. In mathematical terms, PCA defines a set of l vectors of weights or coefficients wk, each of dimension p, which transforms each row vector xi from matrix X to a new vector tk,i in the space represented by the l principal components. The transformation of each xi into a new vector tk,i is calculated as defined in equation (1).(1)tk,i=xi⋅wk,where i=1,…,n and k=1,…,l. Each of the principal components successively captures the maximum possible variance from the original data in matrix X. In order to reduce dimensionality, l<p is usually considered.The data matrixX is decomposed by PCA as T=XW, where W is a weight matrix, of dimensions p×p, and its column vectors correspond to the eigenvectors of the matrix XTX. These eigenvectors turn out to be proportional to the covariance matrix obtained from the data set XT. In other words, PCA diagonalizes the covariance matrix obtained from the data sample. In matrix terms, this can be stated as Q∝XTX=WΛWT, where Λ is the diagonal matrix of eigenvalues of XTX.Notably, PCA transforms a data vectorxi, of dimension p, into new p variables that are uncorrelated in this new space. Given the different levels of variance captured by each component, not all of them need to be preserved. For example, keeping only the first L components (eigenvectors), this results in a truncated version of the transformation TL=XWL, where TL is a matrix of n rows but with only L columns.


### 2.4. Feature Selection Based on Logistic Regression withp Value

The Logistic Regression method is generally used to test the importance or estimate the relationship between a dependent variable, dichotomous binary response, as a function of a single quantitative variable (called Univariate Regression) or of a set of continuous independent variables (called Multivariate Regression) [37]. Regression analysis is a popular statistical process used for modeling and data analysis, indicating significant relationships and impact between the predicted target and the features under study [38]. In a logistic regression model, the evaluation of the fulfillment of the null hypothesis is based on the degree of relationship between the class attribute and each independent attribute of the model, determined by the level of significance and quantified by the p-value [39]. In our work, the null hypothesis corresponds to the nonassociation between the network traffic features and the malware class.In general, the level of significance quantifies the possibility of accepting an erroneous conclusion, that is, of determining that there is an association when in fact there is not [33]. For example, a significance level (usually denoted by α) of 0.05 establishes a 5% risk of accepting a relationship when there is none.In other words, this represents a 95% certainty that the association we are studying is not due to chance. Therefore, if we want to work with a 99% safety margin, it has an implicitp-value of less than 0.01. Therefore, p value ≤α indicates that the association is statistically significant. If p-value ≥α, the association is not statistically significant [40].The formal mathematics underpinning the Logistic Regression method is briefly described in [41] and summarized in the following paragraphs.Letyi and xi,j be the value of the dependent variable and the value of the j-th independent variable j=1,…,k, for the i-th observed data, respectively. The variable yi denotes a binary variable, which determines whether or not the i-th data observed belongs to a given group, being yi=1 when the data belongs to a group, and yi=0 in the case not belong to that group. The probability that yi=1 corresponds to pi. All these variables are formally related as (2)lnp1−p=β0+β1x1+β2x2+⋯+βnxn.In (2) the odds is given by p/1−p and it represents the likelihood that the event will occur. In this context, the natural logarithm of pi/1−pi is equal to the log odds, which allows us to transform a probability in the range 0 to 1 into a value in the range −∞,+∞. In order to isolate the value of p, we raise both sides of the (2) to e, which eliminates the natural logarithm of the left side:(3)p1−p=eβ0+β1x1+β2x2+β3x3+…+βnxn.This expression can be manipulated to isolate the value of p:(4)p=11+e−x,where x stands for β0+β1x1+β2x2+β3x3+⋯+βnxn. This expression turns out to be the Sigmoid or Logistic Function, given by equation (5).(5)Sigmoidx=11+e−x.


## 2.1. Dataset

The CICAndMal2017 dataset is made up of a combination of more than 80 flow time and network packet features to detect and identify malware traffic alongside benign Android applications. This set was built by Lashkari et al. [28] of the Canadian Cybersecurity Institute (CIC). CICAndMal2017 offers 2,126 files in CSV (comma separated values) format and more than 20 gigabytes in PCAP (packet capture) files with malware traffic conversations and benign Android mobile network applications, captured in the years 2015–2017. Network traffic of benign applications are labeled “benign.” Malware traffic is labeled into four categories: adware, ransomware, scareware, and smsware. Each category of malware consists of different families as presented in Table 1 [11]. Originally both sets of CSV and PCAP files are structured with more than 80 Android network traffic features.Table 1
Category and family of malware.
CategoryFamily typeAdwareEdwinKoodousKemogeDowginMobidashYoumiFeiwoSelfmiteRansomwareR ShuanetGooliganChargerPletorLockerPinJisutRansomBOSvpengPornDroidKolerScarewareWannaLockerSimplockerAndroidDefenderFakeAVFakeApp.ALFakeJobOffAVforAndroidPenethoVirusShieldFakeAppSmswareFakeTaoBaoAVpassAndroidSpy.277BeanBotJifakeFakeNotifyBiigeNandroboxFakeInstMazarbotPlanktonFakeMartSMSsniffer


## 2.2. Feature Selection Methods

Two statistical methods of feature selection are described below, known as PCA and Logistic Regression, which were selected and used for their good performance in features selection work in the detection and identification of malware [26, 27, 29–33]. The PCA and logistic regression methods were used to select the most representative features of network traffic from the input data of the CICAndMal2017 dataset.


## 2.3. Feature Selection Based on Principal Component Analysis (PCA)

PCA is a method used to reduce the dimensionality of a large dataset to a smaller one, containing a large part of the information from the original set [24]. Reducing the number of features in a dataset sometimes means losing valuable information, but it also means simplifying the problem, since it is easier to explore and visualize data in small sets [34]. The PCA method therefore allows condensing the information provided by multiple variables into only a few components and having the value of the original variables to calculate these components [35]. Therefore, PCA decomposes a dataset into eigenvectors and eigenvalues. An eigenvector is a direction, for example (x, y), and an eigenvalue is a number that represents the value of the variance in that direction [34]. The main component will be the eigenvector with the highest eigenvalue. There are as many eigenvector/eigenvalue pairs in a dataset as there are dimensions. The eigenvectors do not modify the data, but rather allow us to see them from a different point of view, more related to the internal structure of the data, and with a much more intuitive view of them [30]. Once the eigenvalues, which are a measure of the variance of the data, have been ordered, it is necessary to decide which is the smallest number of eigenvectors or principal components to maintain. To do this, a metric known as explained variance is used, which shows how much variance can be attributed to each of these principal components. Furthermore, as defined in [35], the principal components can be conceptualized as new axes that offer a new coordinate system to evaluate the data, making the differences between the observations in the dataset more visible. The PCA tries to put as much information as possible in the first component, then as much information as possible in the second component, and so on. This process is done until you have a total of principal components equal to the original number of features. As mentioned in [35], there is no single answer or method that allows identifying the optimal number of principal components to select. A very widespread way of proceeding consists of evaluating the proportion of accumulated explained variance and selecting the minimum number of components beyond which the increase is no longer substantial.In other words, PCA corresponds to a linear transformation that takes the input data to a new space of orthogonal axes. In this new space, the axes are ordered such that the first axis captures the largest variance of the original data (called the first principal component), and the last axis captures the smallest variance. Formally [36], let X be a data matrix of dimensions n×p, where each column of data is previously normalized to have zero mean. Here n and p correspond to the number of observations and the number of columns or features of the data set, respectively. In mathematical terms, PCA defines a set of l vectors of weights or coefficients wk, each of dimension p, which transforms each row vector xi from matrix X to a new vector tk,i in the space represented by the l principal components. The transformation of each xi into a new vector tk,i is calculated as defined in equation (1).(1)tk,i=xi⋅wk,where i=1,…,n and k=1,…,l. Each of the principal components successively captures the maximum possible variance from the original data in matrix X. In order to reduce dimensionality, l<p is usually considered.The data matrixX is decomposed by PCA as T=XW, where W is a weight matrix, of dimensions p×p, and its column vectors correspond to the eigenvectors of the matrix XTX. These eigenvectors turn out to be proportional to the covariance matrix obtained from the data set XT. In other words, PCA diagonalizes the covariance matrix obtained from the data sample. In matrix terms, this can be stated as Q∝XTX=WΛWT, where Λ is the diagonal matrix of eigenvalues of XTX.Notably, PCA transforms a data vectorxi, of dimension p, into new p variables that are uncorrelated in this new space. Given the different levels of variance captured by each component, not all of them need to be preserved. For example, keeping only the first L components (eigenvectors), this results in a truncated version of the transformation TL=XWL, where TL is a matrix of n rows but with only L columns.


## 2.4. Feature Selection Based on Logistic Regression withp Value

The Logistic Regression method is generally used to test the importance or estimate the relationship between a dependent variable, dichotomous binary response, as a function of a single quantitative variable (called Univariate Regression) or of a set of continuous independent variables (called Multivariate Regression) [37]. Regression analysis is a popular statistical process used for modeling and data analysis, indicating significant relationships and impact between the predicted target and the features under study [38]. In a logistic regression model, the evaluation of the fulfillment of the null hypothesis is based on the degree of relationship between the class attribute and each independent attribute of the model, determined by the level of significance and quantified by the p-value [39]. In our work, the null hypothesis corresponds to the nonassociation between the network traffic features and the malware class.In general, the level of significance quantifies the possibility of accepting an erroneous conclusion, that is, of determining that there is an association when in fact there is not [33]. For example, a significance level (usually denoted by α) of 0.05 establishes a 5% risk of accepting a relationship when there is none.In other words, this represents a 95% certainty that the association we are studying is not due to chance. Therefore, if we want to work with a 99% safety margin, it has an implicitp-value of less than 0.01. Therefore, p value ≤α indicates that the association is statistically significant. If p-value ≥α, the association is not statistically significant [40].The formal mathematics underpinning the Logistic Regression method is briefly described in [41] and summarized in the following paragraphs.Letyi and xi,j be the value of the dependent variable and the value of the j-th independent variable j=1,…,k, for the i-th observed data, respectively. The variable yi denotes a binary variable, which determines whether or not the i-th data observed belongs to a given group, being yi=1 when the data belongs to a group, and yi=0 in the case not belong to that group. The probability that yi=1 corresponds to pi. All these variables are formally related as (2)lnp1−p=β0+β1x1+β2x2+⋯+βnxn.In (2) the odds is given by p/1−p and it represents the likelihood that the event will occur. In this context, the natural logarithm of pi/1−pi is equal to the log odds, which allows us to transform a probability in the range 0 to 1 into a value in the range −∞,+∞. In order to isolate the value of p, we raise both sides of the (2) to e, which eliminates the natural logarithm of the left side:(3)p1−p=eβ0+β1x1+β2x2+β3x3+…+βnxn.This expression can be manipulated to isolate the value of p:(4)p=11+e−x,where x stands for β0+β1x1+β2x2+β3x3+⋯+βnxn. This expression turns out to be the Sigmoid or Logistic Function, given by equation (5).(5)Sigmoidx=11+e−x.


## 3. Methodology

The work methodology consists of a sequence of four phases that are part of a standard machine learning project [21]:(1)
Analysis and preprocessing(2)
Feature selection(3)
Classifier selection and training(4)
The evaluation of the classifierFigure1 shows these phases. In summary, the methodology first selects two types of datasets, one for each method, then evaluates the different identification algorithms with these datasets. Finally, these results are compared. Each phase of this methodology is described in detail below.Figure 1
Four phases that are part of a standard machine learning project.


### 3.1. Analysis and Preprocessing

The conversations of network traffic of malware and of benign Android applications correspond to the dataset called CICAndMal2017 in CSV format [28]. The idea behind the network conversation level approach delivered by CICAndMal2017 is to present the behavior patterns of network traffic between two or more hosts on the network.Using an application programmed with the use of the Sklearn Python library, the network traffic conversations from the CSV datasets of malware and CSV of benign applications were combined (See Figure1). The total size of the consolidated dataset for this work is 37.8 MB corresponding to 245,138 observations of network traffic for ransomware, adware, scareware, and benign software. In addition, 15 features of network traffic conversations defined in [42] were initially separated (See Table 2). No normalization was applied to the data, since this process was initially carried out in [28]. Also, packets containing TCP (Transmission Control Protocol) retransmissions or other errors are discarded in [28]. The size of our dataset corresponds to 4.4% of the total CICAndMal2017 set.Table 2
Features of network traffic conversations [42].
No.FeatureDescription1Flow durationDuration of the flow in microsecond2Flow byts/sNumber of flow bytes per second3Tot fwd pktsTotal packets in the outgoing4Tot bwd pktsTotal packets in the incoming5Fwd pkt len minMinimum size of packet in outgoing6Fwd pkt len maxMaximum size of packet in outgoing7Fwd pkt len meanMean size of packet in outgoing direction8Fwd pkt len stdStandard deviation size of packets in outgoing9Bwd pkt len minMinimum size of packet in incoming10Bwd pkt len maxMaximum size of packet in incoming11Bwd pkt len meanMean size of packet in incoming direction12Bwd pkt len stdStandard deviation size of packets in incoming13Tot len fwd pktsMinimum length of a flow14Tot len bwd pktsMaximum length of a flow15Mean len pkts/sMean length of a flow16LabelType of malwareA classification can be categorized according to the number of classes to be discriminated. In this form, we can deal with binary classification (two classes, one positive and the other the negative one) or multiclass classification (when the number of classes is more than two). Several issues arise when dealing with multiple classes in classification tasks, mainly the problem of imbalanced classes ([43–47]).Our experiments include data preprocessing for binary and multiclass classification tasks. Specifically, binary classification (malware detection) preprocessing do not require the use of data balancing techniques, because the number of malware and benign application network traffic observations are evenly distributed in the CICAndMal2017 dataset (See Table3). The total malware class traffic corresponds to the aggregation of ransomware, ad-ware and scareware traffic observations.Table 3
Distribution of observations that were used for all classifiers.
TypeMalware trafficBenign trafficRansomwareAdwareScarewareBenignTotal samples41,10040,86639,672123,500Train samples32,88032,69431,73898,800Test samples8,2208,1727,93424,700For the data preprocessing in the multiclass classification task, the CICAndMal2017 data set is divided into four types of classes, that is, the positive classes to be identified as “scareware”, “ransomware” and “adware”; together with the negative class named “benign software”. For the positive classes it is not necessary to balance them since their amounts of network traffic observations are approximately evenly distributed in the data. Due to the fact that the focus of the learning is to discriminate between the different types of malware and, despite the fact that the negative class “benign software” has a ratio of 3 : 1 with respect to each of the positive classes (See Table3), it is decided not to under sample the negative class for balance it with positive classes.


### 3.2. Feature Selection

The selection of features is a fundamental stage in the process of recognition and enumeration of machine learning algorithms patterns, since the vast majority of these algorithms lack metrics that allow them to evaluate the relevance of an attribute for the prediction of the class attribute. Without this prior “filter,” these algorithms can be confused by irrelevant attributes, notoriously deteriorating their performance.The PCA and Logistic Regression (LR) methods, presented in the section Methods and Materials, were used to select the most representative features of network traffic from the input data of the CICAndMal2017 dataset. PCA and LR used the initial 15 network traffic features to create a new subset of mixed features between incoming and outgoing packets and network time streams.


### 3.3. Selection of Classifiers and Training

Six supervised machine learning algorithms were chosen for the classification of network traffic, with the aim of identifying the traffic of malware and benign Android applications. In the literature, the algorithms Random Forest, K-Nearest Neighbors, Decision Tree, Naïve Bayes, Multilayer Perceptron Neural Network and Support Vector Machine, have shown good performance in the classification of network traffic with features of time properties and packet flow (e.g., [11, 28, 42, 48, 49]). In the following, a brief explanation is provided regarding the machine learning algorithms used in this work to estimate the predictive performance of each set of features and for each method used to reduce the dimensionality of the dataset.


#### 3.3.1. Random Forest

Random Forest was proposed by Braimanis in [50]. Random forest is a classifier consisting of a collection of tree-structured classifier defined as [51]:(6)hx,θk,k=1,2,…i,…,where h represents the random forest classifier, the θk are independent identifically distributed random vectors and each tree casts a unit vote for the most popular class at input x [52]. Random forest generates an ensemble of decision trees to classify a new object from an input vector. The input vector is run down each of the trees in the forest. Each tree gives a classification and each tree votes for the class. Regarding the training data, a subset of the data is created for each tree of the forest by using bootstrapping sampling. The chance of overfitting is significantly reduced in comparison to individual decision tree, and there is no need to prune the trees.K-nearest neighbor is a nonparametric classification and regression method [53, 54], where the input considers the k closest training examples in a dataset. In k-NN classification, an input object is classified by a plurality vote of its k nearest neighbors (k > 0 and integer), while in k-NN regression the output is the average of the values of k nearest neighbors. k-NN is a lazy method, where the function is locally approximated and computation is delayed until function evaluation. k-NN relies on the distance computation, where common distance functions can be Euclidean, Manhattan or Minkowski (equations (7)–(9), respectively).Euclidean equation:(7)∑i=1nXi−Yi2.Manhattan equation:(8)∑i=1nXi−Yi.Minkowski equation:(9)∑i=1kXi−Yiq1/q.The training data are vectors in a multidimensional feature space, each one with an associated class label. During training, the algorithm only stores the feature vectors and class labels. During classification, wherek is a user-defined constant, an unlabeled x vector (named a query or test point) is classified by assigning the most frequent label among the k nearest training examples (neighbors) to the given query point. In the case of discrete (nominal or ordinal) variables, Hamming distance can be used. In other domains (e.g. gene expression microarray data) correlation coefficients may be used as a distance metric (e.g., Pearson and Spearman correlation coefficients [55]).


#### 3.3.2. The Decision Tree

The Decision Tree algorithm is a supervised learning approach that builds a predictive model that has a graphical representation. The tree is built by choosing features at the nodes of the tree and arcs associated with the values of the attributes used in the decision tree. In general, the generation of a decision tree is carried out in three stages: selection of features, construction of the tree (its nodes and arcs), and a final stage of pruning the resulting tree [56]. For the experimental process of the decision tree algorithm, the Gini coefficient was used for selection of features [57].The C4.5 algorithm [58] used in this work bases its operation on determining at each step the most predictive attribute with respect to the class attribute, creating a node in the tree for this attribute and dividing the data based on the values of this selected attribute. The division criteria based on this attribute is calculated in the following five steps:First, the expected information required to classify an observation in a data setD, is determined according to the expression shown in equation (10).(10)InfoD=∑i=1mpilog2pi,where pi represents the probability that an observation in the data set D corresponds to the class Ci. In this case, m represents the cardinality of the class attribute.Second, the expected information needed to classify an observation by partitioning the data setD by the v values of an attribute A, is determined according to the expression shown in equation (11).(11)InfoAD=∑j=1vDjD×InfoDj.Thej-th partition j=1,…,v has a weight represented by the term Dj/D.Third, the information gain when the attributeA is used to partition the data set D, is determined according to the expression of equation (12).(12)GainA=InfoD−InfoAD.Fourth, calculate the split information of attributeA with v values, as shown in (13):(13)SplitInfoAD=∑j=1vDjD×log2DjD.Fifth, calculate the gain ratio of attributeA, as shown in (14):(14)GainRatioA=GainASplitInfoAD.In each node of the tree under construction, the attribute with either the highest information gain or gain ratio is selected.


#### 3.3.3. The Naïve Bayes

The Naïve Bayes classifier is based on Bayes’ theorem assuming conditional independence between the independent or predictor variables, given a value of the class attribute (dependent variable). Despite its simplicity, it often shows surprisingly good performance and is widely used, in some cases improving the classification results obtained with more sophisticated methods. Bayes’ theorem provides a method to calculate the posterior probability of the class to which the object to be classified belongs. The Naïve Bayes classifier assumes that the effect of the value of one predictor variable is independent of the values of another predictor variable, given one class value. This assumption is called conditional independence of the class [59].Mathematically, given a vector of featuresX=x1,x2,x3,…,xn and a class variable y, Bayes theorem states that [60](15)Py|X=PX|yPyPX.Thus, the posterior probabilityPy|X is calculated from the likelihood PX|y, the prior probability Py and evidence PX. Then, the term PX|y can be decomposed and simplified, using the chain rule and the conditional independence assumption, resulting in the expression shown in equation (16).(16)Py|X=Px1|yPx2|y…Pxn|yPyPX.In practice, there is interest only in the numerator of the fraction in (16), because the denominator PX does not depend on y and can be considered constant.The Naïve Bayes classifier combines this probability model with a decision rule, in order to select the hypothesis that is most probable, which is known as the Maximum A Posterior (MAP) decision rule. The Bayes classifier applies the function that assigns a class labelc=Ck, for the k value that maximizes the expression shown in equation (17).(17)c=argmaxkpCk∏i=1npxi|Ck,with k=1,…,K


#### 3.3.4. A Multilayer Perceptron Neural Network (MPNN)

A multilayer perceptron neural network (MPNN) refers to a directed neural network formed by several consecutive levels [61]. In an MPNN, during the training process, the input information is propagated from the input layer to the hidden unit layer, and finally reaches the output units to calculate the predicted value. An MPNN seeks to approximate an unknown function, denoted by f∗, such that y=f∗x, where x is the input data and y is the output value calculated by the network. In other words, an MPNN through an iterative process of parameter tuning (parameters denoted by θ) optimizes a loss function to find a mapping f such that y=fx;θ, that is the function f that minimizes the error associated with the loss function. In each unit (neuron), the MPNN performs the calculation indicated by equation (18).(18)y=σW⋅x+b,where y corresponds to the output computed by the neuron, x denotes the vector of input values, W represents the vector of weights of input connections to the neuron, and b corresponds to the bias. σ. denotes the activation function used, usually a nonlinear function. Popular activation functions include the following:(i)
Sigmoid (or logistic):sigmoidx=1/1+e−x(ii)
Hyperbolic tangent:tanhx=ex−e−x/ex+e−x(iii)
Linear unit:ReLUx=maxx,0(iv)
Leaky ReLU:LeakyReLUx=maxαx,x, being α a small constant, e.g., 0.1In this experimental study, the rectified linear unit function (ReLU) is used as activation function.


#### 3.3.5. Support Vector Machines (SVM)

In support vector machines (SVM), a hyperplane that maximizes the margin between two classes in the training data is calculated to perform the classification process. The margin is defined as the minimum perpendicular distance between two points of each class to the separating hyperplane; this hyperplane is fitted during the learning process with the training data or predictors. From these predictors, the vectors that define the hyperplane are selected, which are called support vectors. The optimal hyperplane corresponds to the one that minimizes the training error and, at the same time, has the maximum margin of separation between the two classes. To generalize the cases where the decision limits are not linearly separable, Support Vector Machine projects the training data into another space of higher dimensionality; if the dimensionality of the new space is high enough, the data will always be linearly separable. To avoid having to carry out an explicit projection in a larger dimensional space, a kernel function is used, which implicitly transforms the data to this larger dimensional space to make the linear separation of the classes possible. The kernel function can be polynomial, Gaussian radial basis or sigmoidal perceptron, among others [62].Formally, SVMs are based on the construction of a decision boundary, which takes the form of a hyperplane. In the case of input data that is not clearly linearly separable, kernel functions are used to transform the input data to a new multidimensional space, where a linear decision boundary can be constructed. In either case, the decision function for separating positive from negative classes takes the form of the equation of a hyperplane, as defined by (19) [63].(19)Dx=wϕx+b,where w and b represent the parameters to be found to find the hyperplane that best separates positive from negative examples. Here, ϕx represents the application of the kernel function to transform the original data represented by the vector x into a new space of dimension M. Additionally, Dx/w represents the distance between the hyperplane and the data pattern x. Solving algebraically from (19), the values for the parameters w and b are obtained as indicated in the expressions defined in (20) and (21):(20)w=∑kαkykxk,(21)b=yk−w∗xk.The coefficientsαk are nonzero for the support vectors. It follows from these equations that the parameter w is computed as a linear combination of the training data xk, and the value b is computed as an average of the nonzero αk.


### 3.4. Classifier Performance Evaluation

Usually, the confusion matrix is used to evaluate the performance of the classifiers, since it allows us to analyze and decompose the errors and successes for each value of the class attribute. Three fundamental metrics can be derived from a confusion matrix: precision (P), recall (R) and F-measure (F). These metrics are defined in terms of: true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). In particular, for the network traffic identification process, TP and TN corresponds the number of observations that correctly predict whether it is ransomware or benign application, respectively. On the other hand, FP and FN corresponds to the number of observations that are incorrectly predicted as ransomware or benign application, respectively.(i)
Precision (P) is defined as the ratio of all predicted samples as ransomware traffic that are actually ransomware, and it is computed as shown in equation (22).     (22)P=TPTP+FP.(ii)
Recall (R) is defined as the ratio of all ransomware traffic samples that are expected to be actually ransomware, and it is computed as shown in equation (23).    (23)R=TPTP+FN.(iii)
F-scoreF1: the F1 value corresponds to the harmonic mean of precision and recall values, and therefore it may be better for evaluating performance than overall accuracy. It is computed for the expression shown in equation (24).(24)F1=2×P×RP+R.In addition, the area under the curve (AUC) evaluation metric was used for the receiver operating characteristics (ROC curve). The ROC curve evaluation metric is a graph that shows the performance of a ranking model across all ranking thresholds. A ROC curve represents true positives versus false positives at different classification thresholds [64]. The AUC value corresponds to the two-dimensional area under the entire ROC curve. Thus, the AUC metric provides an aggregated measure of performance at all possible classification thresholds [64], and is calculated as shown in equation (25).(25)AUC=12TPTP+FP+TNTN+FP.To obtain the values of the ROC curves in this work, the benign class was replaced with a value of 0 and the malware class with a value of 1 for the binary classification experiments. Likewise, to obtain the values of the ROC curves in the multiclass classification experiments, the benign class was replaced by the value 0C0, the adware class with the value 1 C1, the scareware class with the value 2 C2 and the ransomware class with the value 3 C3.


## 3.1. Analysis and Preprocessing

The conversations of network traffic of malware and of benign Android applications correspond to the dataset called CICAndMal2017 in CSV format [28]. The idea behind the network conversation level approach delivered by CICAndMal2017 is to present the behavior patterns of network traffic between two or more hosts on the network.Using an application programmed with the use of the Sklearn Python library, the network traffic conversations from the CSV datasets of malware and CSV of benign applications were combined (See Figure1). The total size of the consolidated dataset for this work is 37.8 MB corresponding to 245,138 observations of network traffic for ransomware, adware, scareware, and benign software. In addition, 15 features of network traffic conversations defined in [42] were initially separated (See Table 2). No normalization was applied to the data, since this process was initially carried out in [28]. Also, packets containing TCP (Transmission Control Protocol) retransmissions or other errors are discarded in [28]. The size of our dataset corresponds to 4.4% of the total CICAndMal2017 set.Table 2
Features of network traffic conversations [42].
No.FeatureDescription1Flow durationDuration of the flow in microsecond2Flow byts/sNumber of flow bytes per second3Tot fwd pktsTotal packets in the outgoing4Tot bwd pktsTotal packets in the incoming5Fwd pkt len minMinimum size of packet in outgoing6Fwd pkt len maxMaximum size of packet in outgoing7Fwd pkt len meanMean size of packet in outgoing direction8Fwd pkt len stdStandard deviation size of packets in outgoing9Bwd pkt len minMinimum size of packet in incoming10Bwd pkt len maxMaximum size of packet in incoming11Bwd pkt len meanMean size of packet in incoming direction12Bwd pkt len stdStandard deviation size of packets in incoming13Tot len fwd pktsMinimum length of a flow14Tot len bwd pktsMaximum length of a flow15Mean len pkts/sMean length of a flow16LabelType of malwareA classification can be categorized according to the number of classes to be discriminated. In this form, we can deal with binary classification (two classes, one positive and the other the negative one) or multiclass classification (when the number of classes is more than two). Several issues arise when dealing with multiple classes in classification tasks, mainly the problem of imbalanced classes ([43–47]).Our experiments include data preprocessing for binary and multiclass classification tasks. Specifically, binary classification (malware detection) preprocessing do not require the use of data balancing techniques, because the number of malware and benign application network traffic observations are evenly distributed in the CICAndMal2017 dataset (See Table3). The total malware class traffic corresponds to the aggregation of ransomware, ad-ware and scareware traffic observations.Table 3
Distribution of observations that were used for all classifiers.
TypeMalware trafficBenign trafficRansomwareAdwareScarewareBenignTotal samples41,10040,86639,672123,500Train samples32,88032,69431,73898,800Test samples8,2208,1727,93424,700For the data preprocessing in the multiclass classification task, the CICAndMal2017 data set is divided into four types of classes, that is, the positive classes to be identified as “scareware”, “ransomware” and “adware”; together with the negative class named “benign software”. For the positive classes it is not necessary to balance them since their amounts of network traffic observations are approximately evenly distributed in the data. Due to the fact that the focus of the learning is to discriminate between the different types of malware and, despite the fact that the negative class “benign software” has a ratio of 3 : 1 with respect to each of the positive classes (See Table3), it is decided not to under sample the negative class for balance it with positive classes.


## 3.2. Feature Selection

The selection of features is a fundamental stage in the process of recognition and enumeration of machine learning algorithms patterns, since the vast majority of these algorithms lack metrics that allow them to evaluate the relevance of an attribute for the prediction of the class attribute. Without this prior “filter,” these algorithms can be confused by irrelevant attributes, notoriously deteriorating their performance.The PCA and Logistic Regression (LR) methods, presented in the section Methods and Materials, were used to select the most representative features of network traffic from the input data of the CICAndMal2017 dataset. PCA and LR used the initial 15 network traffic features to create a new subset of mixed features between incoming and outgoing packets and network time streams.


## 3.3. Selection of Classifiers and Training

Six supervised machine learning algorithms were chosen for the classification of network traffic, with the aim of identifying the traffic of malware and benign Android applications. In the literature, the algorithms Random Forest, K-Nearest Neighbors, Decision Tree, Naïve Bayes, Multilayer Perceptron Neural Network and Support Vector Machine, have shown good performance in the classification of network traffic with features of time properties and packet flow (e.g., [11, 28, 42, 48, 49]). In the following, a brief explanation is provided regarding the machine learning algorithms used in this work to estimate the predictive performance of each set of features and for each method used to reduce the dimensionality of the dataset.


### 3.3.1. Random Forest

Random Forest was proposed by Braimanis in [50]. Random forest is a classifier consisting of a collection of tree-structured classifier defined as [51]:(6)hx,θk,k=1,2,…i,…,where h represents the random forest classifier, the θk are independent identifically distributed random vectors and each tree casts a unit vote for the most popular class at input x [52]. Random forest generates an ensemble of decision trees to classify a new object from an input vector. The input vector is run down each of the trees in the forest. Each tree gives a classification and each tree votes for the class. Regarding the training data, a subset of the data is created for each tree of the forest by using bootstrapping sampling. The chance of overfitting is significantly reduced in comparison to individual decision tree, and there is no need to prune the trees.K-nearest neighbor is a nonparametric classification and regression method [53, 54], where the input considers the k closest training examples in a dataset. In k-NN classification, an input object is classified by a plurality vote of its k nearest neighbors (k > 0 and integer), while in k-NN regression the output is the average of the values of k nearest neighbors. k-NN is a lazy method, where the function is locally approximated and computation is delayed until function evaluation. k-NN relies on the distance computation, where common distance functions can be Euclidean, Manhattan or Minkowski (equations (7)–(9), respectively).Euclidean equation:(7)∑i=1nXi−Yi2.Manhattan equation:(8)∑i=1nXi−Yi.Minkowski equation:(9)∑i=1kXi−Yiq1/q.The training data are vectors in a multidimensional feature space, each one with an associated class label. During training, the algorithm only stores the feature vectors and class labels. During classification, wherek is a user-defined constant, an unlabeled x vector (named a query or test point) is classified by assigning the most frequent label among the k nearest training examples (neighbors) to the given query point. In the case of discrete (nominal or ordinal) variables, Hamming distance can be used. In other domains (e.g. gene expression microarray data) correlation coefficients may be used as a distance metric (e.g., Pearson and Spearman correlation coefficients [55]).


### 3.3.2. The Decision Tree

The Decision Tree algorithm is a supervised learning approach that builds a predictive model that has a graphical representation. The tree is built by choosing features at the nodes of the tree and arcs associated with the values of the attributes used in the decision tree. In general, the generation of a decision tree is carried out in three stages: selection of features, construction of the tree (its nodes and arcs), and a final stage of pruning the resulting tree [56]. For the experimental process of the decision tree algorithm, the Gini coefficient was used for selection of features [57].The C4.5 algorithm [58] used in this work bases its operation on determining at each step the most predictive attribute with respect to the class attribute, creating a node in the tree for this attribute and dividing the data based on the values of this selected attribute. The division criteria based on this attribute is calculated in the following five steps:First, the expected information required to classify an observation in a data setD, is determined according to the expression shown in equation (10).(10)InfoD=∑i=1mpilog2pi,where pi represents the probability that an observation in the data set D corresponds to the class Ci. In this case, m represents the cardinality of the class attribute.Second, the expected information needed to classify an observation by partitioning the data setD by the v values of an attribute A, is determined according to the expression shown in equation (11).(11)InfoAD=∑j=1vDjD×InfoDj.Thej-th partition j=1,…,v has a weight represented by the term Dj/D.Third, the information gain when the attributeA is used to partition the data set D, is determined according to the expression of equation (12).(12)GainA=InfoD−InfoAD.Fourth, calculate the split information of attributeA with v values, as shown in (13):(13)SplitInfoAD=∑j=1vDjD×log2DjD.Fifth, calculate the gain ratio of attributeA, as shown in (14):(14)GainRatioA=GainASplitInfoAD.In each node of the tree under construction, the attribute with either the highest information gain or gain ratio is selected.


### 3.3.3. The Naïve Bayes

The Naïve Bayes classifier is based on Bayes’ theorem assuming conditional independence between the independent or predictor variables, given a value of the class attribute (dependent variable). Despite its simplicity, it often shows surprisingly good performance and is widely used, in some cases improving the classification results obtained with more sophisticated methods. Bayes’ theorem provides a method to calculate the posterior probability of the class to which the object to be classified belongs. The Naïve Bayes classifier assumes that the effect of the value of one predictor variable is independent of the values of another predictor variable, given one class value. This assumption is called conditional independence of the class [59].Mathematically, given a vector of featuresX=x1,x2,x3,…,xn and a class variable y, Bayes theorem states that [60](15)Py|X=PX|yPyPX.Thus, the posterior probabilityPy|X is calculated from the likelihood PX|y, the prior probability Py and evidence PX. Then, the term PX|y can be decomposed and simplified, using the chain rule and the conditional independence assumption, resulting in the expression shown in equation (16).(16)Py|X=Px1|yPx2|y…Pxn|yPyPX.In practice, there is interest only in the numerator of the fraction in (16), because the denominator PX does not depend on y and can be considered constant.The Naïve Bayes classifier combines this probability model with a decision rule, in order to select the hypothesis that is most probable, which is known as the Maximum A Posterior (MAP) decision rule. The Bayes classifier applies the function that assigns a class labelc=Ck, for the k value that maximizes the expression shown in equation (17).(17)c=argmaxkpCk∏i=1npxi|Ck,with k=1,…,K


### 3.3.4. A Multilayer Perceptron Neural Network (MPNN)

A multilayer perceptron neural network (MPNN) refers to a directed neural network formed by several consecutive levels [61]. In an MPNN, during the training process, the input information is propagated from the input layer to the hidden unit layer, and finally reaches the output units to calculate the predicted value. An MPNN seeks to approximate an unknown function, denoted by f∗, such that y=f∗x, where x is the input data and y is the output value calculated by the network. In other words, an MPNN through an iterative process of parameter tuning (parameters denoted by θ) optimizes a loss function to find a mapping f such that y=fx;θ, that is the function f that minimizes the error associated with the loss function. In each unit (neuron), the MPNN performs the calculation indicated by equation (18).(18)y=σW⋅x+b,where y corresponds to the output computed by the neuron, x denotes the vector of input values, W represents the vector of weights of input connections to the neuron, and b corresponds to the bias. σ. denotes the activation function used, usually a nonlinear function. Popular activation functions include the following:(i)
Sigmoid (or logistic):sigmoidx=1/1+e−x(ii)
Hyperbolic tangent:tanhx=ex−e−x/ex+e−x(iii)
Linear unit:ReLUx=maxx,0(iv)
Leaky ReLU:LeakyReLUx=maxαx,x, being α a small constant, e.g., 0.1In this experimental study, the rectified linear unit function (ReLU) is used as activation function.


### 3.3.5. Support Vector Machines (SVM)

In support vector machines (SVM), a hyperplane that maximizes the margin between two classes in the training data is calculated to perform the classification process. The margin is defined as the minimum perpendicular distance between two points of each class to the separating hyperplane; this hyperplane is fitted during the learning process with the training data or predictors. From these predictors, the vectors that define the hyperplane are selected, which are called support vectors. The optimal hyperplane corresponds to the one that minimizes the training error and, at the same time, has the maximum margin of separation between the two classes. To generalize the cases where the decision limits are not linearly separable, Support Vector Machine projects the training data into another space of higher dimensionality; if the dimensionality of the new space is high enough, the data will always be linearly separable. To avoid having to carry out an explicit projection in a larger dimensional space, a kernel function is used, which implicitly transforms the data to this larger dimensional space to make the linear separation of the classes possible. The kernel function can be polynomial, Gaussian radial basis or sigmoidal perceptron, among others [62].Formally, SVMs are based on the construction of a decision boundary, which takes the form of a hyperplane. In the case of input data that is not clearly linearly separable, kernel functions are used to transform the input data to a new multidimensional space, where a linear decision boundary can be constructed. In either case, the decision function for separating positive from negative classes takes the form of the equation of a hyperplane, as defined by (19) [63].(19)Dx=wϕx+b,where w and b represent the parameters to be found to find the hyperplane that best separates positive from negative examples. Here, ϕx represents the application of the kernel function to transform the original data represented by the vector x into a new space of dimension M. Additionally, Dx/w represents the distance between the hyperplane and the data pattern x. Solving algebraically from (19), the values for the parameters w and b are obtained as indicated in the expressions defined in (20) and (21):(20)w=∑kαkykxk,(21)b=yk−w∗xk.The coefficientsαk are nonzero for the support vectors. It follows from these equations that the parameter w is computed as a linear combination of the training data xk, and the value b is computed as an average of the nonzero αk.


## 3.3.1. Random Forest

Random Forest was proposed by Braimanis in [50]. Random forest is a classifier consisting of a collection of tree-structured classifier defined as [51]:(6)hx,θk,k=1,2,…i,…,where h represents the random forest classifier, the θk are independent identifically distributed random vectors and each tree casts a unit vote for the most popular class at input x [52]. Random forest generates an ensemble of decision trees to classify a new object from an input vector. The input vector is run down each of the trees in the forest. Each tree gives a classification and each tree votes for the class. Regarding the training data, a subset of the data is created for each tree of the forest by using bootstrapping sampling. The chance of overfitting is significantly reduced in comparison to individual decision tree, and there is no need to prune the trees.K-nearest neighbor is a nonparametric classification and regression method [53, 54], where the input considers the k closest training examples in a dataset. In k-NN classification, an input object is classified by a plurality vote of its k nearest neighbors (k > 0 and integer), while in k-NN regression the output is the average of the values of k nearest neighbors. k-NN is a lazy method, where the function is locally approximated and computation is delayed until function evaluation. k-NN relies on the distance computation, where common distance functions can be Euclidean, Manhattan or Minkowski (equations (7)–(9), respectively).Euclidean equation:(7)∑i=1nXi−Yi2.Manhattan equation:(8)∑i=1nXi−Yi.Minkowski equation:(9)∑i=1kXi−Yiq1/q.The training data are vectors in a multidimensional feature space, each one with an associated class label. During training, the algorithm only stores the feature vectors and class labels. During classification, wherek is a user-defined constant, an unlabeled x vector (named a query or test point) is classified by assigning the most frequent label among the k nearest training examples (neighbors) to the given query point. In the case of discrete (nominal or ordinal) variables, Hamming distance can be used. In other domains (e.g. gene expression microarray data) correlation coefficients may be used as a distance metric (e.g., Pearson and Spearman correlation coefficients [55]).


## 3.3.2. The Decision Tree

The Decision Tree algorithm is a supervised learning approach that builds a predictive model that has a graphical representation. The tree is built by choosing features at the nodes of the tree and arcs associated with the values of the attributes used in the decision tree. In general, the generation of a decision tree is carried out in three stages: selection of features, construction of the tree (its nodes and arcs), and a final stage of pruning the resulting tree [56]. For the experimental process of the decision tree algorithm, the Gini coefficient was used for selection of features [57].The C4.5 algorithm [58] used in this work bases its operation on determining at each step the most predictive attribute with respect to the class attribute, creating a node in the tree for this attribute and dividing the data based on the values of this selected attribute. The division criteria based on this attribute is calculated in the following five steps:First, the expected information required to classify an observation in a data setD, is determined according to the expression shown in equation (10).(10)InfoD=∑i=1mpilog2pi,where pi represents the probability that an observation in the data set D corresponds to the class Ci. In this case, m represents the cardinality of the class attribute.Second, the expected information needed to classify an observation by partitioning the data setD by the v values of an attribute A, is determined according to the expression shown in equation (11).(11)InfoAD=∑j=1vDjD×InfoDj.Thej-th partition j=1,…,v has a weight represented by the term Dj/D.Third, the information gain when the attributeA is used to partition the data set D, is determined according to the expression of equation (12).(12)GainA=InfoD−InfoAD.Fourth, calculate the split information of attributeA with v values, as shown in (13):(13)SplitInfoAD=∑j=1vDjD×log2DjD.Fifth, calculate the gain ratio of attributeA, as shown in (14):(14)GainRatioA=GainASplitInfoAD.In each node of the tree under construction, the attribute with either the highest information gain or gain ratio is selected.


## 3.3.3. The Naïve Bayes

The Naïve Bayes classifier is based on Bayes’ theorem assuming conditional independence between the independent or predictor variables, given a value of the class attribute (dependent variable). Despite its simplicity, it often shows surprisingly good performance and is widely used, in some cases improving the classification results obtained with more sophisticated methods. Bayes’ theorem provides a method to calculate the posterior probability of the class to which the object to be classified belongs. The Naïve Bayes classifier assumes that the effect of the value of one predictor variable is independent of the values of another predictor variable, given one class value. This assumption is called conditional independence of the class [59].Mathematically, given a vector of featuresX=x1,x2,x3,…,xn and a class variable y, Bayes theorem states that [60](15)Py|X=PX|yPyPX.Thus, the posterior probabilityPy|X is calculated from the likelihood PX|y, the prior probability Py and evidence PX. Then, the term PX|y can be decomposed and simplified, using the chain rule and the conditional independence assumption, resulting in the expression shown in equation (16).(16)Py|X=Px1|yPx2|y…Pxn|yPyPX.In practice, there is interest only in the numerator of the fraction in (16), because the denominator PX does not depend on y and can be considered constant.The Naïve Bayes classifier combines this probability model with a decision rule, in order to select the hypothesis that is most probable, which is known as the Maximum A Posterior (MAP) decision rule. The Bayes classifier applies the function that assigns a class labelc=Ck, for the k value that maximizes the expression shown in equation (17).(17)c=argmaxkpCk∏i=1npxi|Ck,with k=1,…,K


## 3.3.4. A Multilayer Perceptron Neural Network (MPNN)

A multilayer perceptron neural network (MPNN) refers to a directed neural network formed by several consecutive levels [61]. In an MPNN, during the training process, the input information is propagated from the input layer to the hidden unit layer, and finally reaches the output units to calculate the predicted value. An MPNN seeks to approximate an unknown function, denoted by f∗, such that y=f∗x, where x is the input data and y is the output value calculated by the network. In other words, an MPNN through an iterative process of parameter tuning (parameters denoted by θ) optimizes a loss function to find a mapping f such that y=fx;θ, that is the function f that minimizes the error associated with the loss function. In each unit (neuron), the MPNN performs the calculation indicated by equation (18).(18)y=σW⋅x+b,where y corresponds to the output computed by the neuron, x denotes the vector of input values, W represents the vector of weights of input connections to the neuron, and b corresponds to the bias. σ. denotes the activation function used, usually a nonlinear function. Popular activation functions include the following:(i)
Sigmoid (or logistic):sigmoidx=1/1+e−x(ii)
Hyperbolic tangent:tanhx=ex−e−x/ex+e−x(iii)
Linear unit:ReLUx=maxx,0(iv)
Leaky ReLU:LeakyReLUx=maxαx,x, being α a small constant, e.g., 0.1In this experimental study, the rectified linear unit function (ReLU) is used as activation function.


## 3.3.5. Support Vector Machines (SVM)

In support vector machines (SVM), a hyperplane that maximizes the margin between two classes in the training data is calculated to perform the classification process. The margin is defined as the minimum perpendicular distance between two points of each class to the separating hyperplane; this hyperplane is fitted during the learning process with the training data or predictors. From these predictors, the vectors that define the hyperplane are selected, which are called support vectors. The optimal hyperplane corresponds to the one that minimizes the training error and, at the same time, has the maximum margin of separation between the two classes. To generalize the cases where the decision limits are not linearly separable, Support Vector Machine projects the training data into another space of higher dimensionality; if the dimensionality of the new space is high enough, the data will always be linearly separable. To avoid having to carry out an explicit projection in a larger dimensional space, a kernel function is used, which implicitly transforms the data to this larger dimensional space to make the linear separation of the classes possible. The kernel function can be polynomial, Gaussian radial basis or sigmoidal perceptron, among others [62].Formally, SVMs are based on the construction of a decision boundary, which takes the form of a hyperplane. In the case of input data that is not clearly linearly separable, kernel functions are used to transform the input data to a new multidimensional space, where a linear decision boundary can be constructed. In either case, the decision function for separating positive from negative classes takes the form of the equation of a hyperplane, as defined by (19) [63].(19)Dx=wϕx+b,where w and b represent the parameters to be found to find the hyperplane that best separates positive from negative examples. Here, ϕx represents the application of the kernel function to transform the original data represented by the vector x into a new space of dimension M. Additionally, Dx/w represents the distance between the hyperplane and the data pattern x. Solving algebraically from (19), the values for the parameters w and b are obtained as indicated in the expressions defined in (20) and (21):(20)w=∑kαkykxk,(21)b=yk−w∗xk.The coefficientsαk are nonzero for the support vectors. It follows from these equations that the parameter w is computed as a linear combination of the training data xk, and the value b is computed as an average of the nonzero αk.


## 3.4. Classifier Performance Evaluation

Usually, the confusion matrix is used to evaluate the performance of the classifiers, since it allows us to analyze and decompose the errors and successes for each value of the class attribute. Three fundamental metrics can be derived from a confusion matrix: precision (P), recall (R) and F-measure (F). These metrics are defined in terms of: true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). In particular, for the network traffic identification process, TP and TN corresponds the number of observations that correctly predict whether it is ransomware or benign application, respectively. On the other hand, FP and FN corresponds to the number of observations that are incorrectly predicted as ransomware or benign application, respectively.(i)
Precision (P) is defined as the ratio of all predicted samples as ransomware traffic that are actually ransomware, and it is computed as shown in equation (22).     (22)P=TPTP+FP.(ii)
Recall (R) is defined as the ratio of all ransomware traffic samples that are expected to be actually ransomware, and it is computed as shown in equation (23).    (23)R=TPTP+FN.(iii)
F-scoreF1: the F1 value corresponds to the harmonic mean of precision and recall values, and therefore it may be better for evaluating performance than overall accuracy. It is computed for the expression shown in equation (24).(24)F1=2×P×RP+R.In addition, the area under the curve (AUC) evaluation metric was used for the receiver operating characteristics (ROC curve). The ROC curve evaluation metric is a graph that shows the performance of a ranking model across all ranking thresholds. A ROC curve represents true positives versus false positives at different classification thresholds [64]. The AUC value corresponds to the two-dimensional area under the entire ROC curve. Thus, the AUC metric provides an aggregated measure of performance at all possible classification thresholds [64], and is calculated as shown in equation (25).(25)AUC=12TPTP+FP+TNTN+FP.To obtain the values of the ROC curves in this work, the benign class was replaced with a value of 0 and the malware class with a value of 1 for the binary classification experiments. Likewise, to obtain the values of the ROC curves in the multiclass classification experiments, the benign class was replaced by the value 0C0, the adware class with the value 1 C1, the scareware class with the value 2 C2 and the ransomware class with the value 3 C3.


## 4. Experiments and Results

In this section, the experimental results in feature selection and performance evaluation are presented. Firstly, in selection of features, the results of the experiments carried out by the PCA and Logistic Regression methods are presented, to reduce and select the set of network features most representative of the behavior of malware and benign Android applications traffic (See Table2).Secondly, in performance evaluation, the experiments and results of the empirical evaluation of the performance of the following supervised algorithms are presented: Naïve Bayes, support vector machine, multilayer perceptron neural network, decision tree, random forest and K-nearest neighbors, with the purpose of identifying malware traffic from the features selected by the PCA and Logistic Regression statistical methods. All experiments were executed on Microsoft Windows 10 Professional (64 bit) with a second-generation Intel Core i7 2.20 Ghz processor and 16 GB of RAM. The Python 3.7.0 programming language was used to perform the data preprocessing tasks, the selection of features and the construction of the classification models. For the classifiers, the default parameters of Python scikit-learn were utilized.


### 4.1. Experimental Results of the Selection of Features

The experiment carried out with the PCA method calculated the proportion of explained variance for each computed component (See Figure2.) and the accumulated explained variance (See Figure 3.) derived from the initial dataset (See Table 2). The 94% total variability is explained by the first 9 PCA components. The results of the PCA technique discarded 6 of 15 components related to the flow in bytes and flow mean length packets per second (Flow Byts/s, Mean Len Pkts/s), the average size of input and output packets (Bwd Pkt Len Mean, Fwd Pkt Len Mean), and the standard input and output packet size (Bwd Pkt Len Std, Fwd Pkt Len Std). Therefore, the PCA method presents 9 features that are representative of the network traffic of malware and benign Android applications from the initial dataset (See Table 4).Figure 2
The variance explained by each component computed by the PCA method.Figure 3
The accumulated explained variance associated to the PCA components.Table 4
Features that are most representative of the network traffic of malware and benign for the PCA method.
NO.FeatureDescription1Flow durationDuration of the flow in microsecond2Tot fwd pktsTotal packets in the outgoing3Tot bwd pktsTotal packets in the incoming4Fwd pkt len minMinimum size of packet in outgoing5Fwd pkt len maxMaximum size of packet in outgoing6Bwd pkt len minMinimum size of packet in incoming7Bwd pkt len maxMaximum size of packet in incoming8Tot len fwd pktsMinimum length of a flow9Tot len bwd pktsMaximum length of a flow10LabelType of malwareRegarding the results obtained by the Logistic Regression method, Table5 presents a summary of the results obtained from the execution of the first experiment with p-value of 0.05, where the minimum length features of input packets (Bwd_Pkt_Len_Min) and the standard deviation of the input packet length (Bwd_Pkt_Len_Std) presented a statistically nonsignificant association with respect to the class of the Logistic Regression model. For the second Logistic Regression experiment, the two variables with the lowest significance (Bwd_Pkt_Len_Min and Bwd_Pkt_Len_Std) obtained from the first experiment with p-value ≤0.05 were removed, and the significance value restriction was lowered to p-value ≤0.01. The results of the second experiment did not find differences with respect to the model of features selected by the first experiment of p-value ≤0.05. Table 6 presents the 13 most representative features of the network traffic of malware and benign Android applications selected by the Logistic Regression method.Table 5
Summary of the results obtained from the execution of the first experiment withp-value ≤0.05.
FeatureCoefOdds ratiop valueConst−0.43240.652820.001Flow_Duration0.000001.000000.001Tot_Fwd_Pkts0.000001.000000.001Tot_Bwd_Pkts−0.077040.925850.001TotLen_Fwd_Pkts0.000161.000160.001TotLen_Bwd_Pkts0.000051.000050.001Fwd_Pkt_Len_Min0.006491.006520.001Bwd_Pkt_Len_Min−0.000090.999910.192Fwd_Pkt_Len_Max−0.002170.997840.001Bwd_Pkt_Len_Max−0.00027−0.000270.001Fwd_Pkt_Len_Mean−0.007980.992050.001Bwd_Pkt_Len_Mean0.000391.000390.001Fwd_Pkt_Len_Std0.010841.010900.001Bwd_Pkt_Len_Std0.000201.000200.100Mean len Pkts/s0.000001.000000.001Flow_Byts/s0.000001.000000.001Table 6
Features that are most representative of the network traffic of malware and benign for logistic regression.
NO.FeatureDescription1Flow durationDuration of the flow in microsecond2Flow Byts/sNumber of flow bytes per second,3Tot fwd pktsTotal packets in the outgoing4Tot bwd pktsTotal packets in the incoming5Fwd pkt len minMinimum size of packet in outgoing6Fwd pkt len maxMaximum size of packet in outgoing7Fwd pkt len meanMean size of packet in outgoing direction8Fwd pkt len StdStandard deviation size of packets in outgoing9Bwd pkt len maxMaximum size of packet in incoming10Bwd pkt len meanMean size of packet in incoming direction11Tot len fwd pktsMinimum length of a flow12Tot len bwd pktsMaximum length of a flow13Mean len Pkts/sMean length of a flow14LabelType of malware


### 4.2. Experimental Results for the Performance Evaluation of Supervised Algorithms

With the network traffic features already selected by the PCA and Logistic Regression methods, and considering the initial dataset, two experimental scenarios were defined to evaluate the performance of the six supervised algorithms for the task of classifying malware traffic. These are binary classification and multiclass classification. The binary classification scenario includes observations of network traffic with class-tagged malware and benign software. The multiclass classification scenario includes four types of classes: scareware, ransomware, adware, and benign software. For these two scenarios, the ratio of the training and testing set is 80 : 20. For both scenarios, experiments with and without N-fold cross-validation were performed. In N-fold cross-validation the dataset is randomly partitioned between N observations and the evaluations are executed by N iterations. In each iteration, N-1 sets of samples are selected for training and the other one is left to validate the precision of the classifier [65]. N = 10 was selected to carry out the experiments, according to the N-fold performance obtained in studies related to the detection and identification of malware ([28, 57, 66, 67]). Likewise, for both scenarios, the initial dataset (CDI), the dataset with features selected by PCA (CDPCA) and the dataset with features selected by Logistic Regression (CDLR) were considered. Tables 7–11, and 12 present the experimental results obtained through the combination of the initial features, the features selected by the PCA and LR methods, together with the application of the six supervised algorithms mentioned (DT, NB, RF, SVM, MPNN, and KNN).Table 7
Binary classification results without cross validation.
ModelPrecisionRecallF1 scoreAccuracyAUCCDI + DT0.940.940.9494.03%0.94CDI + NB0.560.510.3851.40%0.64CDI + RF0.950.950.9595.30%0.97CDI + SVM0.860.820.8282.26%0.86CDI + MPNN0.780.750.7474.63%0.65CDI + KNN0.890.890.8989.36%0.93CDPCA + DT0.910.910.9190.90%0.93CDPCA + NB0.620.510.6351.06%0.66CDPCA + RF0.940.940.9493.76%0.96CDPCA + SVM0.860.820.8282.26%0.86CDPCA + MPNN0.710.700.6969.53%0.61CDPCA + KNN0.880.880.8887.61%0.92CDLR + DT0.940.940.9494,02%0.94CDLR + NB0.560.510.3851.40%0.64CDLR + RF0.960.960,9696.42%0.98CDLR + SVM0.860.820.8282.26%0.86CDLR + MPNN0.710.700.7070.20%0.61CDLR + KNN0.890.890.8989.36%0.93Table 8
Binary classification results using cross validation withN = 10.
ModelPrecisionRecallF1 scoreAccuracy (%)AUCCDI + DT0.940.940.9494.050.94CDI + NB0.560.510.3851.390.64CDI + RF0.950.950.9595.290.97CDI + SVM0.870.870.8786.860.86CDI + MPNN0.710.580.5158.370.45CDI + KNN0.890.890.8989.360.93CDPCA + DT0.940.940.9493.980.94CDPCA + NB0.560.510.3851.390.64CDPCA + RF0.940.940.9493.760.96CDPCA + SVM0.870.870.8786.860.86CDPCA + MPNN0.680.650.6465.340.55CDPCA + KNN0.880.880.8887.610.92CDLR + DT0.940.940.9493.990.94CDLR + NB0.560.510.3851.390.64CDLR + RF0.960.960.9696.410.98CDLR + SVM0.860.820.8282.260.86CDLR + MPNN0.790.780.7777.500.79CDLR + KNN0.890.890.8989.410.93Table 9
Results for the multiclass classification scenario without cross validation.
ModelPrecisionRecallF1 scoreAccuracy (%)CDI + DT0.860.860.8585.7CDI + NB0.460.500.3549.76CDI + RF0.860.860.8686.93CDI + SVM0.780.780.7777.53CDI + MPNN0.610.550.5658.4CDI + KNN0.620.630.6262.59CDPCA + DT0.850.860.8585.67CDPCA + NB0.470.510.3550.63CDPCA + RF0.860.860.8685.63CDPCA + SVM0.780.780.7777.53CDPCA + MPNN0.580.590.6258.91CDPCA + KNN0.610.630.6262.52CDLR + DT0.850.860.8585.66CDLR + NB0.460.500.3549.76CDLR + RF0.870.870.8787.06CDLR + SVM0.780.780.7777.53CDLR + MPNN0.550.590.5459.45CDLR + KNN0.630.620.6362.53Table 10
The ROC curve final mixture of the performance results with multiclass without cross validation.
ModelC0C1C2C3CDI + DT0.970.960.940.96CDI + NB0.620.640.610.72CDI + RF0.970.980.950.97CDI + SVM0.550.770.560.67CDI + MPNN0.550.770.560.67CDI + KNN0.720.940.730.77CDPCA + DT0.970.980.950.97CDPCA + NB0.650.660.630.74CDPCA + RF0.960.970.940.97CDPCA + SVM0.550.770.560.67CDPCA + MPNN0.720.820.700.72CDPCA + KNN0.730.930.700.79CDLR + DT0.970.960.940.96CDLR + NB0.620.640.610.72CDLR + RF0.970.980.950.97CDLR + SVM0.550.770.560.67CDLR + MPNN0.970.980.950.97CDLR + KNN0.720.940.730.77Table 11
Results for the multiclass classification with cross validation withN = 10.
ModelPrecisionRecallF1 scoreAccuracy (%)CDI + DT0.860.860.8585.7CDI + NB0.420.500.3549.60CDI + RF0.870.870.8786.92CDI + SVM0.780.780.7777.53CDI + MPNN0.520.570.4856.57CDI + KNN0.710.710.7171.46CDPCA + DT0.850.850.8585.47CDPCA + NB0.470.510.3550.54CDPCA + RF0.860.860.8685.63CDPCA + SVM0.780.780.7777.53CDPCA + MPNN0.550.590.5358.73CDPCA + KNN0.700.710.7071.15CDLR + DT0.850.860.8585.50CDLR + NB0.420.500.3549.60CDLR + RF0.870.870.8787.05CDLR + SVM0.780.780.7777.53CDLR + MPNN0.560.590.5158.57CDLR + KNN0.710.710.7171.43Table 12
The ROC curve final mixture of the performance results with multiclass cross validation withN = 10.
ModelC0C1C2C3CDI + DT0.970.960.940.96CDI + NB0.640.670.630.70CDI + RF0.970.980.950.97CDI + SVM0.640.670.630.70CDI + MPNN0.550.770.560.67CDI + KNN0.720.940.730.77CDPCA + DT0.970.980.950.97CDPCA + NB0.650.700.650.71CDPCA + RF0.960.970.940.97CDPCA + SVM0.550.770.560.67CDPCA + MPNN0.720.820.700.72CDPCA + KNN0.730.930.700.79CDLR + DT0.970.960.940.96CDLR + NB0.640.670.630.70CDLR + RF0.970.980.950.97CDLR + SVM0.550.770.560.67CDLR + MPNN0.970.980.950.97CDLR + KNN0.720.940.730.77In the binary classification experiment without Cross-validation (Table7) it was found that, using the combination of features of logistic regression and random forest (CDLR + RF), the best performance was obtained, with an average precision rate of 0.96, recall of the 0.96, F1 value of 0.96, accuracy of 96.42% and AUC of 0.98 with respect to the rest of the experiments (See Figure 4). Likewise, for the binary classification with Cross-validation N = 10 (See Tables 8 and Figure 5), the combination of logistic regression and random forest features (CDLR + RF) obtained a slightly lower accuracy performance of 96.41%, a precision of 0.96, a recall of 0.96 and F1 value of 0.96. Among both experiments of binary classification with initial features, selected by PCA and logistic regression, Naïve Bayes obtained the worst performance (see Figures 4 and 5). For the multiclass classification scenario without cross-validation (see Tables 9, 10 and Figure 6) it was found that, for the combination of the features selected by the Logistic Regression method and the Random Forest algorithm (CDLR + RF), the best performance was obtained. In this case, an average precision of 0.87 was obtained, recall of 0.87, F1 value of 0.87, accuracy of 87.06% and an AUC average rate greater than 85% for the malware classification. Likewise, for the multiclass classification with Cross-validation N = 10 (See Tables 11, 12 and Figure 7), the combination of Logistic Regression and Random Forest features (CDLR + RF) obtained a slightly lower accuracy performance with 87.05%, equating the average precision with 0.87, recall of 0.87 and F1 value of 0.87. Among both multiclass classification experiments with initial features, selected by PCA and logistic regression, Naïve Bayes obtained the worst performance (See Figures 6 and 7). The ROC curve (see Tables 10, 12 and Figure 8) summarizes the final mixture of the performance results of each of the classifiers combined with the features obtained by the logistic regression method. The ROC curve shows a good initial discriminative ability, losing it smoothly at the rate of 90% true positive (TP). Random forest presented the highest AUC for the 13 most representative features selected from Logistic Regression (CDLR + RF).Figure 4
Experimental result of malware binary classification without cross-validation.Figure 5
Experimental result of malware binary classification with cross-validationN = 10.Figure 6
Experimental result of malware multiclass classification without cross-validation.Figure 7
Experimental result of malware Multiclass classification with cross-validationN = 10.Figure 8
Performance results of each of the classifiers combined with the 13 features obtained by the logistic regression method.


## 4.1. Experimental Results of the Selection of Features

The experiment carried out with the PCA method calculated the proportion of explained variance for each computed component (See Figure2.) and the accumulated explained variance (See Figure 3.) derived from the initial dataset (See Table 2). The 94% total variability is explained by the first 9 PCA components. The results of the PCA technique discarded 6 of 15 components related to the flow in bytes and flow mean length packets per second (Flow Byts/s, Mean Len Pkts/s), the average size of input and output packets (Bwd Pkt Len Mean, Fwd Pkt Len Mean), and the standard input and output packet size (Bwd Pkt Len Std, Fwd Pkt Len Std). Therefore, the PCA method presents 9 features that are representative of the network traffic of malware and benign Android applications from the initial dataset (See Table 4).Figure 2
The variance explained by each component computed by the PCA method.Figure 3
The accumulated explained variance associated to the PCA components.Table 4
Features that are most representative of the network traffic of malware and benign for the PCA method.
NO.FeatureDescription1Flow durationDuration of the flow in microsecond2Tot fwd pktsTotal packets in the outgoing3Tot bwd pktsTotal packets in the incoming4Fwd pkt len minMinimum size of packet in outgoing5Fwd pkt len maxMaximum size of packet in outgoing6Bwd pkt len minMinimum size of packet in incoming7Bwd pkt len maxMaximum size of packet in incoming8Tot len fwd pktsMinimum length of a flow9Tot len bwd pktsMaximum length of a flow10LabelType of malwareRegarding the results obtained by the Logistic Regression method, Table5 presents a summary of the results obtained from the execution of the first experiment with p-value of 0.05, where the minimum length features of input packets (Bwd_Pkt_Len_Min) and the standard deviation of the input packet length (Bwd_Pkt_Len_Std) presented a statistically nonsignificant association with respect to the class of the Logistic Regression model. For the second Logistic Regression experiment, the two variables with the lowest significance (Bwd_Pkt_Len_Min and Bwd_Pkt_Len_Std) obtained from the first experiment with p-value ≤0.05 were removed, and the significance value restriction was lowered to p-value ≤0.01. The results of the second experiment did not find differences with respect to the model of features selected by the first experiment of p-value ≤0.05. Table 6 presents the 13 most representative features of the network traffic of malware and benign Android applications selected by the Logistic Regression method.Table 5
Summary of the results obtained from the execution of the first experiment withp-value ≤0.05.
FeatureCoefOdds ratiop valueConst−0.43240.652820.001Flow_Duration0.000001.000000.001Tot_Fwd_Pkts0.000001.000000.001Tot_Bwd_Pkts−0.077040.925850.001TotLen_Fwd_Pkts0.000161.000160.001TotLen_Bwd_Pkts0.000051.000050.001Fwd_Pkt_Len_Min0.006491.006520.001Bwd_Pkt_Len_Min−0.000090.999910.192Fwd_Pkt_Len_Max−0.002170.997840.001Bwd_Pkt_Len_Max−0.00027−0.000270.001Fwd_Pkt_Len_Mean−0.007980.992050.001Bwd_Pkt_Len_Mean0.000391.000390.001Fwd_Pkt_Len_Std0.010841.010900.001Bwd_Pkt_Len_Std0.000201.000200.100Mean len Pkts/s0.000001.000000.001Flow_Byts/s0.000001.000000.001Table 6
Features that are most representative of the network traffic of malware and benign for logistic regression.
NO.FeatureDescription1Flow durationDuration of the flow in microsecond2Flow Byts/sNumber of flow bytes per second,3Tot fwd pktsTotal packets in the outgoing4Tot bwd pktsTotal packets in the incoming5Fwd pkt len minMinimum size of packet in outgoing6Fwd pkt len maxMaximum size of packet in outgoing7Fwd pkt len meanMean size of packet in outgoing direction8Fwd pkt len StdStandard deviation size of packets in outgoing9Bwd pkt len maxMaximum size of packet in incoming10Bwd pkt len meanMean size of packet in incoming direction11Tot len fwd pktsMinimum length of a flow12Tot len bwd pktsMaximum length of a flow13Mean len Pkts/sMean length of a flow14LabelType of malware


## 4.2. Experimental Results for the Performance Evaluation of Supervised Algorithms

With the network traffic features already selected by the PCA and Logistic Regression methods, and considering the initial dataset, two experimental scenarios were defined to evaluate the performance of the six supervised algorithms for the task of classifying malware traffic. These are binary classification and multiclass classification. The binary classification scenario includes observations of network traffic with class-tagged malware and benign software. The multiclass classification scenario includes four types of classes: scareware, ransomware, adware, and benign software. For these two scenarios, the ratio of the training and testing set is 80 : 20. For both scenarios, experiments with and without N-fold cross-validation were performed. In N-fold cross-validation the dataset is randomly partitioned between N observations and the evaluations are executed by N iterations. In each iteration, N-1 sets of samples are selected for training and the other one is left to validate the precision of the classifier [65]. N = 10 was selected to carry out the experiments, according to the N-fold performance obtained in studies related to the detection and identification of malware ([28, 57, 66, 67]). Likewise, for both scenarios, the initial dataset (CDI), the dataset with features selected by PCA (CDPCA) and the dataset with features selected by Logistic Regression (CDLR) were considered. Tables 7–11, and 12 present the experimental results obtained through the combination of the initial features, the features selected by the PCA and LR methods, together with the application of the six supervised algorithms mentioned (DT, NB, RF, SVM, MPNN, and KNN).Table 7
Binary classification results without cross validation.
ModelPrecisionRecallF1 scoreAccuracyAUCCDI + DT0.940.940.9494.03%0.94CDI + NB0.560.510.3851.40%0.64CDI + RF0.950.950.9595.30%0.97CDI + SVM0.860.820.8282.26%0.86CDI + MPNN0.780.750.7474.63%0.65CDI + KNN0.890.890.8989.36%0.93CDPCA + DT0.910.910.9190.90%0.93CDPCA + NB0.620.510.6351.06%0.66CDPCA + RF0.940.940.9493.76%0.96CDPCA + SVM0.860.820.8282.26%0.86CDPCA + MPNN0.710.700.6969.53%0.61CDPCA + KNN0.880.880.8887.61%0.92CDLR + DT0.940.940.9494,02%0.94CDLR + NB0.560.510.3851.40%0.64CDLR + RF0.960.960,9696.42%0.98CDLR + SVM0.860.820.8282.26%0.86CDLR + MPNN0.710.700.7070.20%0.61CDLR + KNN0.890.890.8989.36%0.93Table 8
Binary classification results using cross validation withN = 10.
ModelPrecisionRecallF1 scoreAccuracy (%)AUCCDI + DT0.940.940.9494.050.94CDI + NB0.560.510.3851.390.64CDI + RF0.950.950.9595.290.97CDI + SVM0.870.870.8786.860.86CDI + MPNN0.710.580.5158.370.45CDI + KNN0.890.890.8989.360.93CDPCA + DT0.940.940.9493.980.94CDPCA + NB0.560.510.3851.390.64CDPCA + RF0.940.940.9493.760.96CDPCA + SVM0.870.870.8786.860.86CDPCA + MPNN0.680.650.6465.340.55CDPCA + KNN0.880.880.8887.610.92CDLR + DT0.940.940.9493.990.94CDLR + NB0.560.510.3851.390.64CDLR + RF0.960.960.9696.410.98CDLR + SVM0.860.820.8282.260.86CDLR + MPNN0.790.780.7777.500.79CDLR + KNN0.890.890.8989.410.93Table 9
Results for the multiclass classification scenario without cross validation.
ModelPrecisionRecallF1 scoreAccuracy (%)CDI + DT0.860.860.8585.7CDI + NB0.460.500.3549.76CDI + RF0.860.860.8686.93CDI + SVM0.780.780.7777.53CDI + MPNN0.610.550.5658.4CDI + KNN0.620.630.6262.59CDPCA + DT0.850.860.8585.67CDPCA + NB0.470.510.3550.63CDPCA + RF0.860.860.8685.63CDPCA + SVM0.780.780.7777.53CDPCA + MPNN0.580.590.6258.91CDPCA + KNN0.610.630.6262.52CDLR + DT0.850.860.8585.66CDLR + NB0.460.500.3549.76CDLR + RF0.870.870.8787.06CDLR + SVM0.780.780.7777.53CDLR + MPNN0.550.590.5459.45CDLR + KNN0.630.620.6362.53Table 10
The ROC curve final mixture of the performance results with multiclass without cross validation.
ModelC0C1C2C3CDI + DT0.970.960.940.96CDI + NB0.620.640.610.72CDI + RF0.970.980.950.97CDI + SVM0.550.770.560.67CDI + MPNN0.550.770.560.67CDI + KNN0.720.940.730.77CDPCA + DT0.970.980.950.97CDPCA + NB0.650.660.630.74CDPCA + RF0.960.970.940.97CDPCA + SVM0.550.770.560.67CDPCA + MPNN0.720.820.700.72CDPCA + KNN0.730.930.700.79CDLR + DT0.970.960.940.96CDLR + NB0.620.640.610.72CDLR + RF0.970.980.950.97CDLR + SVM0.550.770.560.67CDLR + MPNN0.970.980.950.97CDLR + KNN0.720.940.730.77Table 11
Results for the multiclass classification with cross validation withN = 10.
ModelPrecisionRecallF1 scoreAccuracy (%)CDI + DT0.860.860.8585.7CDI + NB0.420.500.3549.60CDI + RF0.870.870.8786.92CDI + SVM0.780.780.7777.53CDI + MPNN0.520.570.4856.57CDI + KNN0.710.710.7171.46CDPCA + DT0.850.850.8585.47CDPCA + NB0.470.510.3550.54CDPCA + RF0.860.860.8685.63CDPCA + SVM0.780.780.7777.53CDPCA + MPNN0.550.590.5358.73CDPCA + KNN0.700.710.7071.15CDLR + DT0.850.860.8585.50CDLR + NB0.420.500.3549.60CDLR + RF0.870.870.8787.05CDLR + SVM0.780.780.7777.53CDLR + MPNN0.560.590.5158.57CDLR + KNN0.710.710.7171.43Table 12
The ROC curve final mixture of the performance results with multiclass cross validation withN = 10.
ModelC0C1C2C3CDI + DT0.970.960.940.96CDI + NB0.640.670.630.70CDI + RF0.970.980.950.97CDI + SVM0.640.670.630.70CDI + MPNN0.550.770.560.67CDI + KNN0.720.940.730.77CDPCA + DT0.970.980.950.97CDPCA + NB0.650.700.650.71CDPCA + RF0.960.970.940.97CDPCA + SVM0.550.770.560.67CDPCA + MPNN0.720.820.700.72CDPCA + KNN0.730.930.700.79CDLR + DT0.970.960.940.96CDLR + NB0.640.670.630.70CDLR + RF0.970.980.950.97CDLR + SVM0.550.770.560.67CDLR + MPNN0.970.980.950.97CDLR + KNN0.720.940.730.77In the binary classification experiment without Cross-validation (Table7) it was found that, using the combination of features of logistic regression and random forest (CDLR + RF), the best performance was obtained, with an average precision rate of 0.96, recall of the 0.96, F1 value of 0.96, accuracy of 96.42% and AUC of 0.98 with respect to the rest of the experiments (See Figure 4). Likewise, for the binary classification with Cross-validation N = 10 (See Tables 8 and Figure 5), the combination of logistic regression and random forest features (CDLR + RF) obtained a slightly lower accuracy performance of 96.41%, a precision of 0.96, a recall of 0.96 and F1 value of 0.96. Among both experiments of binary classification with initial features, selected by PCA and logistic regression, Naïve Bayes obtained the worst performance (see Figures 4 and 5). For the multiclass classification scenario without cross-validation (see Tables 9, 10 and Figure 6) it was found that, for the combination of the features selected by the Logistic Regression method and the Random Forest algorithm (CDLR + RF), the best performance was obtained. In this case, an average precision of 0.87 was obtained, recall of 0.87, F1 value of 0.87, accuracy of 87.06% and an AUC average rate greater than 85% for the malware classification. Likewise, for the multiclass classification with Cross-validation N = 10 (See Tables 11, 12 and Figure 7), the combination of Logistic Regression and Random Forest features (CDLR + RF) obtained a slightly lower accuracy performance with 87.05%, equating the average precision with 0.87, recall of 0.87 and F1 value of 0.87. Among both multiclass classification experiments with initial features, selected by PCA and logistic regression, Naïve Bayes obtained the worst performance (See Figures 6 and 7). The ROC curve (see Tables 10, 12 and Figure 8) summarizes the final mixture of the performance results of each of the classifiers combined with the features obtained by the logistic regression method. The ROC curve shows a good initial discriminative ability, losing it smoothly at the rate of 90% true positive (TP). Random forest presented the highest AUC for the 13 most representative features selected from Logistic Regression (CDLR + RF).Figure 4
Experimental result of malware binary classification without cross-validation.Figure 5
Experimental result of malware binary classification with cross-validationN = 10.Figure 6
Experimental result of malware multiclass classification without cross-validation.Figure 7
Experimental result of malware Multiclass classification with cross-validationN = 10.Figure 8
Performance results of each of the classifiers combined with the 13 features obtained by the logistic regression method.


## 5. Discussion

Some of the features selected by Logistic Regression withp-value were expected according to the state of the art. The duration of flows in microseconds, the number of byte flows per second, the total of packets in the outbound direction and the total of packets in the inbound direction, are well known to researchers in malware detection and identification in network traffic. Logistic Regression presented features like those recorded in previous studies in the related work section and the authors’ prior knowledge. The network flow time variables Flow Byts/s and flow Mean Len Pkts/s discarded by PCA are associated with a higher probability of identifying malware and differentiating it from benign applications. The features selected by Logistic Regression obtained good results, especially when the Random Forest and Decision Tree algorithms were used in binary and multiclass classification. In [28] a good performance of Random Forest (RF), Decision Tree (DT) and K-Nearest Neighbor (KNN) was also achieved with an average precision of 85% and recall of 88% in binary classification. However, RF, DT and KNN presented an average precision and recall of less than 49% in multiclass classification. In [28], 9 features and the CfsSubsetEval, Best First and Infogain methods were used. The selection of features through Logistic Regression allowed to obtain representative features for the identification of malware, avoiding the shadowing of features generated by the PCA method. As presented in Tables 9–11 and 12, the results obtained by the proposed method are optimistic towards the selection of new network traffic features. Likewise, Random Forest achieved promising results in binary and competitive classification compared to the results presented by other works in multiclass classification Tables 13 and 14. This is especially true when there are a greater number of features related to network flow times.Table 13
Result of performance evaluation and comparison metrics in binary classification.
Results worksComparison metricsPrecisionRecallF1 scoreAccuracyAUCLashkari et al. [28]0.850.850.8585%N/AAbuthawabeh et al. [11]0.860.860.8686%N/AMurtaz et al. [48]N/AN/AN/AN/AN/AAbuthawabeh et al. [49]N/AN/AN/A87%N/AChen et al. [42]0.950.950.9595%N/AProposed method0.960.960.9696%0.98Table 14
Result of performance evaluation and comparison metrics in multiclass classification.
Results worksComparison metricsPrecisionRecallF1 scoreAccuracy (%)AUCLashkari et al. [28]0.490.490.4949N/AAbuthawabeh et al. [11]0.790.790.7979N/AMurtaz et al. [48]N/AN/AN/A94N/AAbuthawabeh et al. [49]0.890.890.8989N/AChen et al. [42]0.840.840.8484N/AProposed method0.870.870.87870.98


## 6. Related Work

In this section, related works are discussed and its analysis is logically divided into two parts. Firstly, we review related works about classification task to detect and identify android network malware in general. Secondly, we discuss related works about classification task to detect and identify android network malware that use specifically the CICAndMal2017 dataset.


### 6.1. Classification of Android Network Malware

The selection of features is key in most cases of classification and regression tasks and impacts the performance of machine learning algorithms, in particular this is valid in the detection and identification of malware-type network traffic. First of all, the task of identifying malicious traffic (binary classification) is important, for which selecting the relevant characteristics and discarding the redundant ones have a significant impact on the construction and application of machine learning models. Secondly, it is also very important to identify the type of malware (multiclass classification) that is present in traffic identified as malicious, in order to better understand and resolve the potential attack [68].Mainly, it is possible to distinguish between machine learning applications that detect versus those that identify malware traffic based on the type of output of the classification problem. A malware detection system generates as output a binary value or a value in the range between 0 and 1,y=fx, that is, an output value y before an input vector x. In the case of a malware identification system, the output is associated with a probability of belonging to a class or family of malicious traffic, being y∈ℝN, where N is the number of different families [23].Among the different amount of studies facing malware detection and identification in networks, there are some works that deal with feature selection prior to testing detection and classification techniques.In [69], the analysis of computer network traffic based on a system that detects the presence of malware is exposed. A total of 972 behavioral characteristics of network traffic over the Internet, at the transport and application level, were extracted and analyzed. The selection of the subset of features was based on the correlation feature selection algorithm, which fed three classification algorithms: random forest, Naive Bayes and decision tree (specifically the J48 algorithm).In [70], 9 traffic features were selected to improve the efficiency of a network traffic classifier, which implements a mobile malware traffic detector. This subset of features was selected using CfsSubsetEval attribute evaluator and the Best First search method. In order to characterize malware families, the proposed model uses features including flow-based, packet-based and time-based features. The results obtained by using the proposed feature set reach an accuracy above 93% in the detection of malware. In addition, a 92% of success probability on characterization and on average a false positive rate less than 0.08 percent are obtained. These performance values are required in the implementation of a system operating on real world malware detection.In [71] the authors proposed a dynamic analysis technique in Android Malware detection.Initially, the data obtained is related to memory and CPU usage, packet transfer and system calls, which were considered as input to the feature extraction task. Second, the Gain Ratio Attribute Evaluator algorithm was used to select features. Third, the APKPure and Genome Project datasets were used to perform the classifier training and validation process to discriminate between malicious and benign traffic. The results obtained indicate that, using the Random Forest algorithm,91.7%, 93.1% and 90% of global accuracy, precision and recall, respectively, are obtained.In Hernandez and Goseva -Popstojanova [72], the authors focused on malware detection based on the use of characteristics extracted from network traffic and system logs. features were used. Experimental work was carried out based on four algorithms (Naive Bayes, J48, Random Forest and PART) for the malware detection task. In determining the least number of characteristics, information gain was used as a metric to rank the attributes. Based on the F1 score and G-score metrics, the classifiers with the best performance turned out to be those obtained with the J48 and PART algorithms. Remarkably, the J48 algorithm obtained a similar performance using only the 5 best features than in the case of using all 88 original features, which translates into a decrease in computational cost during training. In the case of the PART algorithm, similar results were obtained when using the 14 best features versus using all 88 original features.Wang et al. [73] proposed an efficient malware detection method using the text semantics of HTTP network traffic with NPL (natural language processing), chi-square algorithm to automatically select the best features, and an SVM machine learning linear classifier. In the evaluation, 31,706 benign streams and 5,258 malicious streams were used, and the proposed classifier outperforms existing approaches and obtains an accuracy of 99.15%.Shabtai et al. [74] contributed a system that detects malicious behavior through network traffic analysis. This is done by logging user-specific network traffic patterns per examined app and subsequently identifying deviations that can be flagged as malicious. To evaluate their model, the C4.5 algorithm is employed, achieving an accuracy of up to 94%.


### 6.2. Classification of Android Network Malware Using the CICAndMal2017 Dataset

Some selected related works [11, 28, 42, 48, 49] associated with the detection and identification of malware Android in network traffic based on classification task of machine learning, which consider the selection of features in the CICAndMal2017 dataset, are presented in Table 15. This table shows the total number of benign software and malware network traffic conversations, the feature selection method, and the number of features selected for the classification process for each related job. For this work, the number of 15 network features obtained from Chen et al. [42] and the set of benign software and malware traffic conversations acquired from the 2,216 CSV files in the work of Lashkari et al. [28], were used.Table 15
Total number of benign software and malware network traffic conversations for each related job.
WorksN°TrafficConversationMethodFeature selectedLashkari et al. [28]5.494CfsSubsetEval9Best firstInfogainAbuthawabeh et al. [11]244.594Random forest14Recursive featureElimination, light GBMMurtaz et al. [48]126.391Data gain9Cfs subsetSVM wekaAbuthawabeh et al. [49]305.743Random forest9Recursive featureElimination, light GBMChen et al. [42]244.802Python method15In 2018 Lashkari et al. [28] present a systematic approach to generate real Android mobile traffic using CICAndMal2017. Also [28] proposed an experimental strategy of binary and multiclass classification, together with three classifiers, carrying out the training and evaluation of their performance based on the CICAndMal2017 dataset. The results of [28] for binary classification show an average precision of 85% and recall of 88% for the random forest (RF), K-nearest neighbor (KNN) and decision tree (DT) algorithms. However, random forest (RF), K-nearest neighbor (KNN) and decision tree (DT) presented an average precision and recall of less than 49% in multiclass classification.Abuthawabeh et al. [11], present an improved model for the detection, categorization, and classification of malware families in network traffic using CICAndMal2017. The authors use the enhanced PeerShark tool for 14 feature extraction and an assembly with three feature selection algorithms to achieve choosing the 9 most representative features from the dataset. The feature selection algorithms are RF, Recursive Feature Elimination (RFE), and Light GBM. The model developed in [11] was trained and evaluated using three classifiers: RF, KNN and DT. The study by Abuthawabeh et al. [11], compared the results of the improved model with the model of Lashkari et al. [28], through precision and recall metrics, obtaining slightly better results in binary detection and a significant improvement in multiclass classification, over an average greater than 79% in precision and recall.In [42] the malware identification work based on Android network traffic analysis in the CICAndMal2017 dataset is presented. The authors selected a PCAP file from each family of benign malware and software to build the customized dataset. The chosen conversations were taken at random. Features were extracted from PCAP files by two steps. The first step was developed using a Java program to separate the network flows. Then 15 features were extracted, using a Python program. Three supervised machine learning classifiers were used: RF, KNN, and DT. In [42] a binary (malware and benign) and multiclass experimental strategy was used with three categories: Adware, Ransomware and Scareware. The authors use three metrics to evaluate the performance of the classifiers: precision, recall and measure F. For the binary classification of malware, the results show that the random forest classifier achieved the highest results with 92% of the measure F and a 95% accuracy and recall. The rest of the classifiers obtained more than 85% of all the metrics used. For the multiclass classification of malware, the RF, KNN and DT classifiers achieved an average of more than 80% in each of the metrics selected by the authors. As in binary classification, Random Forest achieved the highest results in multiclass classification with an average of 84% precision, recall and measure F.In [48], a framework for the detection and classification of Android malware is proposed in the CICandMal2017 dataset. An experimental multiclass classification strategy was proposed with network traffic from benign applications, adware, and general malware. Weka’s Data Gain, CFSSubset and Support Vector Machine (SVM) feature selection algorithms were used. The CFSSubset algorithm selected the 9 most significant features for the framework presented by [48]. The results presented indicate that, for the random forest (RF), K-nearest neighbor (KNN), decision tree (DT), random tree (RT) and LOGISTIC REGRESSION (LR) classifiers, an accuracy of 94% was obtained. The authors do not show precision and recall results.In [49], the authors propose a model to detect and categorize malware based on network traffic features considering the CICAndMal2017 dataset. The 9 most significant features were chosen using the assembly technique through three feature selection algorithms: random forest, recursive feature elimination (RFE) and light GBM. Likewise, the model was evaluated with three classifiers: random forest (RF), decision tree (DT) and extra tree (ET). The experimental results show that the selected features improved the detection and categorization of Android malware. The extra tree (ET) algorithm obtained the best accuracy with 87.75%, precision of 89.35% and a recall of 85.33% for binary classification. For multiclass classification, extra tree also obtained the best performance with 79.7% accuracy, 80.24% precision and 79.3% recall. Using the same dataset (CICAndMal2017) in [49], Manzano et al. in [75] study the classification between benign applications and ransomware using only three classificators (RF, DT, and KNN), without a focus on the feature selection. In [75], the authors conclude that the selection of features can help differentiate ransomware from the traffic of benign applications.In summary, the results obtained by these experimental works will provide a baseline of comparison for binary and multiclass classification for the network traffic data considered. In terms of the feature selection method, this work performs an exhaustive exploration based on two feature reduction methods: principal component analysis (PCA) and logistic regression (LR). Both are combined with six traditional learning machine learning algorithms (RF, KNN, DT, NB, MLP, and SVM), building subsets of 10 and 13 features for PCA and LR, respectively. In terms of performance of the models generated by these algorithms, the proposed method exhibits better results than the related works [11, 28, 42, 48, 49] reviewed, in binary classification, and superior to the same works in terms of precision and recall for multiclass classification. The exception is that reported in [48], where only the global multiclass accuracy is presented, without reporting other relevant metrics such as precision, recall and the F measure, both for the case of binary and multiclass classification.


## 6.1. Classification of Android Network Malware

The selection of features is key in most cases of classification and regression tasks and impacts the performance of machine learning algorithms, in particular this is valid in the detection and identification of malware-type network traffic. First of all, the task of identifying malicious traffic (binary classification) is important, for which selecting the relevant characteristics and discarding the redundant ones have a significant impact on the construction and application of machine learning models. Secondly, it is also very important to identify the type of malware (multiclass classification) that is present in traffic identified as malicious, in order to better understand and resolve the potential attack [68].Mainly, it is possible to distinguish between machine learning applications that detect versus those that identify malware traffic based on the type of output of the classification problem. A malware detection system generates as output a binary value or a value in the range between 0 and 1,y=fx, that is, an output value y before an input vector x. In the case of a malware identification system, the output is associated with a probability of belonging to a class or family of malicious traffic, being y∈ℝN, where N is the number of different families [23].Among the different amount of studies facing malware detection and identification in networks, there are some works that deal with feature selection prior to testing detection and classification techniques.In [69], the analysis of computer network traffic based on a system that detects the presence of malware is exposed. A total of 972 behavioral characteristics of network traffic over the Internet, at the transport and application level, were extracted and analyzed. The selection of the subset of features was based on the correlation feature selection algorithm, which fed three classification algorithms: random forest, Naive Bayes and decision tree (specifically the J48 algorithm).In [70], 9 traffic features were selected to improve the efficiency of a network traffic classifier, which implements a mobile malware traffic detector. This subset of features was selected using CfsSubsetEval attribute evaluator and the Best First search method. In order to characterize malware families, the proposed model uses features including flow-based, packet-based and time-based features. The results obtained by using the proposed feature set reach an accuracy above 93% in the detection of malware. In addition, a 92% of success probability on characterization and on average a false positive rate less than 0.08 percent are obtained. These performance values are required in the implementation of a system operating on real world malware detection.In [71] the authors proposed a dynamic analysis technique in Android Malware detection.Initially, the data obtained is related to memory and CPU usage, packet transfer and system calls, which were considered as input to the feature extraction task. Second, the Gain Ratio Attribute Evaluator algorithm was used to select features. Third, the APKPure and Genome Project datasets were used to perform the classifier training and validation process to discriminate between malicious and benign traffic. The results obtained indicate that, using the Random Forest algorithm,91.7%, 93.1% and 90% of global accuracy, precision and recall, respectively, are obtained.In Hernandez and Goseva -Popstojanova [72], the authors focused on malware detection based on the use of characteristics extracted from network traffic and system logs. features were used. Experimental work was carried out based on four algorithms (Naive Bayes, J48, Random Forest and PART) for the malware detection task. In determining the least number of characteristics, information gain was used as a metric to rank the attributes. Based on the F1 score and G-score metrics, the classifiers with the best performance turned out to be those obtained with the J48 and PART algorithms. Remarkably, the J48 algorithm obtained a similar performance using only the 5 best features than in the case of using all 88 original features, which translates into a decrease in computational cost during training. In the case of the PART algorithm, similar results were obtained when using the 14 best features versus using all 88 original features.Wang et al. [73] proposed an efficient malware detection method using the text semantics of HTTP network traffic with NPL (natural language processing), chi-square algorithm to automatically select the best features, and an SVM machine learning linear classifier. In the evaluation, 31,706 benign streams and 5,258 malicious streams were used, and the proposed classifier outperforms existing approaches and obtains an accuracy of 99.15%.Shabtai et al. [74] contributed a system that detects malicious behavior through network traffic analysis. This is done by logging user-specific network traffic patterns per examined app and subsequently identifying deviations that can be flagged as malicious. To evaluate their model, the C4.5 algorithm is employed, achieving an accuracy of up to 94%.


## 6.2. Classification of Android Network Malware Using the CICAndMal2017 Dataset

Some selected related works [11, 28, 42, 48, 49] associated with the detection and identification of malware Android in network traffic based on classification task of machine learning, which consider the selection of features in the CICAndMal2017 dataset, are presented in Table 15. This table shows the total number of benign software and malware network traffic conversations, the feature selection method, and the number of features selected for the classification process for each related job. For this work, the number of 15 network features obtained from Chen et al. [42] and the set of benign software and malware traffic conversations acquired from the 2,216 CSV files in the work of Lashkari et al. [28], were used.Table 15
Total number of benign software and malware network traffic conversations for each related job.
WorksN°TrafficConversationMethodFeature selectedLashkari et al. [28]5.494CfsSubsetEval9Best firstInfogainAbuthawabeh et al. [11]244.594Random forest14Recursive featureElimination, light GBMMurtaz et al. [48]126.391Data gain9Cfs subsetSVM wekaAbuthawabeh et al. [49]305.743Random forest9Recursive featureElimination, light GBMChen et al. [42]244.802Python method15In 2018 Lashkari et al. [28] present a systematic approach to generate real Android mobile traffic using CICAndMal2017. Also [28] proposed an experimental strategy of binary and multiclass classification, together with three classifiers, carrying out the training and evaluation of their performance based on the CICAndMal2017 dataset. The results of [28] for binary classification show an average precision of 85% and recall of 88% for the random forest (RF), K-nearest neighbor (KNN) and decision tree (DT) algorithms. However, random forest (RF), K-nearest neighbor (KNN) and decision tree (DT) presented an average precision and recall of less than 49% in multiclass classification.Abuthawabeh et al. [11], present an improved model for the detection, categorization, and classification of malware families in network traffic using CICAndMal2017. The authors use the enhanced PeerShark tool for 14 feature extraction and an assembly with three feature selection algorithms to achieve choosing the 9 most representative features from the dataset. The feature selection algorithms are RF, Recursive Feature Elimination (RFE), and Light GBM. The model developed in [11] was trained and evaluated using three classifiers: RF, KNN and DT. The study by Abuthawabeh et al. [11], compared the results of the improved model with the model of Lashkari et al. [28], through precision and recall metrics, obtaining slightly better results in binary detection and a significant improvement in multiclass classification, over an average greater than 79% in precision and recall.In [42] the malware identification work based on Android network traffic analysis in the CICAndMal2017 dataset is presented. The authors selected a PCAP file from each family of benign malware and software to build the customized dataset. The chosen conversations were taken at random. Features were extracted from PCAP files by two steps. The first step was developed using a Java program to separate the network flows. Then 15 features were extracted, using a Python program. Three supervised machine learning classifiers were used: RF, KNN, and DT. In [42] a binary (malware and benign) and multiclass experimental strategy was used with three categories: Adware, Ransomware and Scareware. The authors use three metrics to evaluate the performance of the classifiers: precision, recall and measure F. For the binary classification of malware, the results show that the random forest classifier achieved the highest results with 92% of the measure F and a 95% accuracy and recall. The rest of the classifiers obtained more than 85% of all the metrics used. For the multiclass classification of malware, the RF, KNN and DT classifiers achieved an average of more than 80% in each of the metrics selected by the authors. As in binary classification, Random Forest achieved the highest results in multiclass classification with an average of 84% precision, recall and measure F.In [48], a framework for the detection and classification of Android malware is proposed in the CICandMal2017 dataset. An experimental multiclass classification strategy was proposed with network traffic from benign applications, adware, and general malware. Weka’s Data Gain, CFSSubset and Support Vector Machine (SVM) feature selection algorithms were used. The CFSSubset algorithm selected the 9 most significant features for the framework presented by [48]. The results presented indicate that, for the random forest (RF), K-nearest neighbor (KNN), decision tree (DT), random tree (RT) and LOGISTIC REGRESSION (LR) classifiers, an accuracy of 94% was obtained. The authors do not show precision and recall results.In [49], the authors propose a model to detect and categorize malware based on network traffic features considering the CICAndMal2017 dataset. The 9 most significant features were chosen using the assembly technique through three feature selection algorithms: random forest, recursive feature elimination (RFE) and light GBM. Likewise, the model was evaluated with three classifiers: random forest (RF), decision tree (DT) and extra tree (ET). The experimental results show that the selected features improved the detection and categorization of Android malware. The extra tree (ET) algorithm obtained the best accuracy with 87.75%, precision of 89.35% and a recall of 85.33% for binary classification. For multiclass classification, extra tree also obtained the best performance with 79.7% accuracy, 80.24% precision and 79.3% recall. Using the same dataset (CICAndMal2017) in [49], Manzano et al. in [75] study the classification between benign applications and ransomware using only three classificators (RF, DT, and KNN), without a focus on the feature selection. In [75], the authors conclude that the selection of features can help differentiate ransomware from the traffic of benign applications.In summary, the results obtained by these experimental works will provide a baseline of comparison for binary and multiclass classification for the network traffic data considered. In terms of the feature selection method, this work performs an exhaustive exploration based on two feature reduction methods: principal component analysis (PCA) and logistic regression (LR). Both are combined with six traditional learning machine learning algorithms (RF, KNN, DT, NB, MLP, and SVM), building subsets of 10 and 13 features for PCA and LR, respectively. In terms of performance of the models generated by these algorithms, the proposed method exhibits better results than the related works [11, 28, 42, 48, 49] reviewed, in binary classification, and superior to the same works in terms of precision and recall for multiclass classification. The exception is that reported in [48], where only the global multiclass accuracy is presented, without reporting other relevant metrics such as precision, recall and the F measure, both for the case of binary and multiclass classification.


## 7. Conclusion and Future Work

This work presents the results of an empirical evaluation of the performance of six supervised algorithms: Naïve Bayes, support vector machine, multilayer perceptron neural network, decision tree and K-nearest neighbors, to identify malware traffic, considering two statistical methods of reduction and selection of features of the Android network traffic dataset CICAndMal2017. First, the PCA and Logistic Regression feature selection methods were run, extracting the most representative features for the identification of malware and benign Android applications. For the PCA method, the first 9 components obtained 94% of the variance of the data, being the most representative candidate features between network input and output packets. However, the PCA experiments evidenced a shading of network flow time-type features, which were evidenced as a significant contribution to the Logistic Regression method. This is the case of the network flow time variables Flow Byts/s and Mean Len Pkts/s discarded by the PCA. PCA-based feature selection performed the worst on the accuracy metrics for binary and multiclass classification. The logistic regression algorithm was able to detect a more accurate and useful feature correlation weight for binary and multiclass classification experiments. Logistic regression provided the features that contributed to obtaining the best binary and multi-classification results, with and without cross-validation, using the random forest algorithm. Although PCA managed to reduce the initial set of features to have better performance of the algorithms used, Logistic Regression managed to return with itsp-value score <0.05 those network flow time variables, to improve the general precision of the models, both binary and multiclass.The experimental classification results show that the network traffic classification technique based on random forest obtained the best identification of malware traffic and benign traffic with an average accuracy of 96% and an AUC of 0.98, over the rest of the binary classification algorithms. In addition, random forest had the best average malware accuracy performance at 87% and AUC average over all other multiclass classification algorithms. The lowest malware classification result around the binary and multiclass classification scenarios was the Naïve Bayes algorithm. Future work will consider improving the rate of identification of malware and benign applications through experiments based on cross-validation with different N-fold settings. Class balancing methods will be addressed to achieve a more efficient work on malware classification.Finally, as the evolution of Android malware attacks is rapid and permanent, the features of our dataset may not be practical for detecting new malware cases. Therefore, applying deep learning methods can be good alternatives for the detection and identification of the traffic of new cases of malware, since these approaches do not depend on predefined features, but are built internally, as part of the complex and hierarchical process of deep learning.



---

*Source: 6760920-2022-04-07.xml*