# MidSiot: A Multistage Intrusion Detection System for Internet of Things

Authors: Nguyen Dat-Thinh; Ho Xuan-Ninh; Le Kim-Hung
Journal: Wireless Communications and Mobile Computing
 (2022)

---

## Abstract

Internet of Things (IoT) has been thriving in recent years, playing an important role in a multitude of various domains, including industry 4.0, smart transportation, home automation, and healthcare. As a result, a massive number of IoT devices are deployed to collect data from our surrounding environment and transfer these data to other systems over the Internet. This may lead to cybersecurity threats, such as denial of service attacks, brute-force attacks, and unauthorized accesses. Unfortunately, many IoT devices lack solid security mechanisms and hardware security supports because of their limitations in computational capability. In addition, the heterogeneity of devices in IoT networks causes nontrivial challenges in detecting security threats. In this article, we present a collaborative intrusion detection system (IDS), namely, MidSiot, deployed at both Internet gateways and IoT local gateways. Our proposed IDS consists of three stages: (1) classifying the type of each IoT device in the IoT network; (2) differentiating between benign and malicious network traffic; and (3) identifying the type of attacks targeting IoT devices. The last two stages are handled by the Internet gateways, whereas the first stage is on the local gateway to leverage the computational resources from edge devices. The evaluation results on three popular IDS datasets (IoTID20, CIC-IDS-2017, and BOT-IoT) indicate our proposal could detect seven common cyberattacks targeting IoT devices with an average accuracy of 99.68% and outperforms state-of-the-art IDSs. This demonstrates that MidSiot could be an effective and practical IDS to protect IoT networks.


---

## Body

## 1. Introduction

The number of devices connecting to the Internet has been growing at a breathtaking pace over the past decades. From two billion in 2006, it reached 200 billion in 2020 because of the proliferation of mobile computing and the Internet of Things (IoT) [1]. As a result, these devices play a critical role in primary industries (e.g., healthcare, manufacturing, retailing, security, and transportation) by providing intelligent services, such as tracking inventory, managing machines, monitoring patient health, and detecting abnormality. They are anticipated to boost the total global worth of IoT to 6.2 trillion dollars by 2025, most of which come from manufacturing (2.3 trillion dollars) and healthcare (2.5 trillion dollars) [2]. It is apparent that IoT is the driving force of evolution in every daily aspect.However, ensuring security and privacy for the IoT devices is a nontrivial challenge due to their limitation in computational capability, which is insufficient for traditional security mechanisms. This makes them susceptible to wide-ranging cyberattacks, such as data leakage, spoofing, and DoS/DDoS. In a report published by Kaspersky, the first half of 2021 witnessed 1.5 billion attacks against smart devices aiming at stealing data, mining cryptocurrency, or building botnets [3]. In September 2016, an infamous attack performed by Mirai malware turned 380,000 devices into botnets that launched DDoS attacks against several services and organizations, including Dyn-a domain registration service provider [4]. Moreover, this malware is capable of mutating [5]. On 12 December 2017, its variant exploited a zero-day flaw in Huawei HG532 routers to speed up its infection. One year later, the number of variants was increased significantly, such as Okiru, Masuta, OMG, Wicked, Hakai, and Yowai [6].To eliminate such security threats, an intrusion detection system (IDS) is commonly deployed at network gateways. It constantly monitors network traffics coming from various sources to detect abnormalities, which may be security threats. Following [7], the attack detection approaches of the IDS are categorized into signature-based and anomaly-based. The former approach identifies cyberattacks by comparing a set of signatures (or rules) extracted from known attacks with incoming traffic. We note that the key difference between IoT network traffic and other network traffics is the diversity and volume. The diversity of IoT network traffic comes from the heterogeneity of IoT devices and their communication protocols, resulting in diverse network behaviors. Furthermore, IoT devices exponentially increase and generate massive data traversing the Internet. Due to these characteristics, the rule-based detection mechanism, SNORT is a typical example, is insufficient for IoT networks. In detail, SNORT is ineffective for complex attacks signs of which are various and implicit in network traffic. In addition, SNORT needs to maintain a large rule-set and security experts in the loop to update these rules frequently. Therefore, it is not efficient and scalable enough for IoT scenarios. The latter approach, which is the most popular, makes use of machine learning (ML) to construct a model of normal network traffic patterns. This model is then used to measure the similarity between the incoming traffic and known patterns to detect malicious traffic. Although the anomaly-based approach using machine learning considerably alleviates the weaknesses of the signature-based approach, it still has several limitations.(i)
Neglecting a collaborative edge-cloud architecture: Training and inferring tasks in machine learning are resource-intensive, so they are usually handled by cloud platforms in existing work. This might decrease the detection performance because the network traffic at the cloud level, which is merged from several gateways consisting of various data sources, is intricate. In contrast, edge devices are resource-constraint IoT devices that are insufficient to handle complex machine learning tasks (e.g., detecting abnormal activities, training detection models). Offloading these tasks on edge devices severely affects other services running on these devices. However, we believe that these devices could handle specific tasks to increase the IDS’s detection performance regardless of their limitation in computation capability. Therefore, a collaborative edge-cloud architecture for instruction detection systems is necessary to overcome this limitation.(ii)
Lacking IoT device-type identification: Because the IoT device types are various and heterogeneous, their network behaviors are highly diverse. For example, the high UDP packet rate coming from IoT cameras is normal, but the one from temperature sensors is a sign of a security threat. This may lead to false attack detection. Thus, identifying device types and considering them as an input feature of the attack detection model is crucial to increase the detection accuracy of IDSs.(iii)
Detecting a limited set of attack types: Existing works about IDS are extensive but primarily concerned with detecting a limited set of attacks in a general domain, such as DoS and spoofing. Given the rising prevalence of IoT, there is an essential need to address a larger set of attacks targeting IoT networks.To solve the above limitation, we present MidSiot, a machine learning-based three-stage IDS designed for IoT networks supporting collaboration between local gateways and Internet gateways of Internet Service Provider (ISP) (solving the first limitation). In more detail, to leverage edge computing and enhance the attack detection accuracy, the first stage is operated at local gateways to identify IoT devices based on their behaviors in the network (solving the first limitation), whereas the next two stages powered by a machine learning model are handled by the Internet gateways to not only differentiate between normal and malicious network traffic, but also accurately identify the seven common attack types (solving the first limitation). The evaluation results on existing IDS datasets for the IoT domain (IoTID20, BOT-IoT, and CIC-IDS-2017) show that MidSiot could detect seven popular attacks targeting IoT devices with an average accuracy of 99.68%. Our main contributions presented in this study are as follows:(1)
A collaborative architecture for IoT IDSs to leverage the computational resources of edge gateway to enhance IDS’s detection performance.(2)
A lightweight and robust machine learning-based IDS constituting of three stages to accurately detect various cyberattacks pointing at IoT devices.(3)
We intensively evaluate our proposal on popular IDS datasets and examine the resampling techniques to address imbalanced datasets during our experiments.The remainder of the article is organized as follows. In Section2, we discuss related work. The MidSiot architecture and its detection method are presented in Section 3. Section 4 reports the evaluation of our method through IDS datasets, and we conclude our work in Section 5.


## 2. Related Works

In recent years, there has been an increased interest in exploring machine learning for enhancing the detection quality of IDSs [17, 18]. In [9], the authors proposed an anomaly detection mechanism using a single machine learning classifier. The authors of [10] presented a scalable k-NN-based online anomaly detection addressing the lazy-learning problem in wireless sensor networks [10]. The works in [16] also employed anomaly detection techniques for IDSs using binary classification. This means that they cannot identify the type of attack. In [11], the authors proposed an ensemble of autoencoders for online IDS whose performance is comparable to offline anomaly detectors. Ref. [12] is a novel approach for IDSs in which the authors applied convolutional neural network to predict the attack types. 98% accuracy on the NSL-KDD dataset was achieved in this experiment. There are also hybrid-IDSs where anomaly-based and signature-based approaches are used to develop the IDS. Such a typical system is introduced in [19], in which packet header anomaly detection, network traffic anomaly detection, and SNORT are combined. In [13], the authors leveraged four machine learning algorithms to derive rule-sets used as signatures for their IDS. In [8], a hierarchical architecture including multiple neural networks was proposed to detect malicious packets and identify the attack types hidden inside these packets. The authors in [14] introduced a hierarchical structure for IDS that separates the detection process into different steps. The authors in [20] suggested a distributed architecture for smart home IDSs that offload complex tasks onto the Internet Service Provider (ISP) and deliver simple ones to the smart home gateway [20].In terms of datasets used for IDSs, the authors in [21] proposed a new dataset called IoTID20, which was also evaluated in their work by implementing several machine learning algorithms (e.g., logistic regression, decision tree, random forest), which results in increasing F1-score for both binary classification and multiclass classification. The authors in [15] developed a new realistic botnet dataset for use in IoT networks, and as a result, it mainly consists of DDoS attacks. Another dataset is CICIDS-2017 including attack traffic generated by their testbed and realistic background traffic created by the B-profile system [22]. All of these 3 datasets were constructed using the CICFlowMeter tool (formerly known as ISCXFlowMeter), thereby having a similar set of features. Regarding resampling methods for network intrusion detection system (NIDS), the literature at [23] compared multiple undersampling techniques for NIDS on CICIDS-2017 and CICIDS-2018 datasets, including random, cluster centroids, and nearmiss algorithms. The authors concluded that these undersampling methods reduced models’ training time, and K-nearest neighbor has the most significant improvement. There are also other works that implemented a combination of several resampling techniques, such as oversampling and undersampling [24]. Their experimental results showed that the oversampling method increases the training time, whereas the undersampling method decreases this time. In addition, if the dataset is highly imbalanced, these methods improve the recall score notably. The authors in [25] proposed an algorithm-level class balancing technique that addresses the underlying issue about attack class imbalance in IDS datasets, resulting in identifying various attack categories with better accuracy than the CNN models.In recent years, many researchers have geared toward blockchain applications in intrusion detection systems thanks to its potential in protecting data integrity and privacy. The authors in [26] investigated the challenges and limitations of blockchain to intrusion detection in addition to their applications, such as the overhead traffic with limited handling capability of intrusion detection and extensive energy and cost usage of blockchain. Despite these difficulties, blockchain still has the potential to mitigate the data sharing and trust management issues in collaborative intrusion detection. As far as collaborative intrusion detection systems are concerned, a series of research [27–29] provided blockchain challenge-based collaborative intrusion detections. In these systems, the authors leveraged the strength of blockchain to investigate the trust mechanism in a network of IDS nodes. Their goals are to enhance the robustness of trust management against attacks as well as to protect the alarm aggregation process from malicious inputs. The works in [30, 31] made some contributions in the same direction, but the authors specifically targeted intrusion detection systems in a software-defined network. The authors in [32] proposed a deep blockchain framework to offer security-based distributed intrusion detection and privacy-based blockchain with smart contracts in IoT networks. Although the experimental results of the intrusion detection system were optimistic, the classification algorithm in use was a bidirectional long short-term memory, which accompanying blockchain might aggregate more computational burden on operating IoT devices.To summarize the related works, Table1 presents the state-of-the-art intrusion detection systems and their characteristics, including targeting security threats, attack detection method, evaluation datasets, attack-type and device-type detection, and lightweight. We can see that none of them is lightweight enough to classify the type of attack and its target. In addition, several approaches are evaluated by non-IoT datasets or testbeds having a small number of IoT devices. Thus, previous IDS proposals are insufficient for deploying to practical IoT ecosystem.Table 1
Summary of current works on Intrusion Detection Systems for Internet of Things.
WorkSecurity threatDetection methodValidation datasetAttack-type detectionDevice-type detectionLightweightZhang et al. [8]DoS, R2L, U2R, and PROBEDeep learningKDD Cup 1999 DataYesNo—Wang and Stolfo [9]58 attack types with 1999 DARPA dataset CUCS dataset (Code Red II, Buffer overflow)1-gram models1999 DARPA IDS Dataset CUCS DatasetYesNo—Xie et al. [10]—Machine learningReal WSN data sets—NoYesMirsky et al. [11]Recon., MITM, DoS, BotnetAutoencoderReal-testbedYesNoYesInce [12]DoS, probe, R2L, U2RDeep learningNSL-KDDYesNo—Kumar et al. [13]Dos, exploit, probe, genericHybridUNSW-NB15YesNo—Anthi et al. [14]Attack reconnaissances, DoS attacks, man-in-the-middle attacks, replay attacks, DNS spoofingMachine learningReal-testbedYesYes—Koroniotis et al. [15]DoS/DDoS attacks, keylogging, data theftDeep learningBOT-IoT DatasetYesNo—Liu et al. [16]Vulnerability scanners, ARP spoofing, DoS attacks, Mirai BotnetMachine learningIOTID-20 DatasetNoNoYesProposed SystemScanning methods (Host Discovery, Port scanning, OS/Version Detection) ARP Spoofing, SYN Flooding, Host Discovery, Telnet Bruce-force, UDP/ACK/HTTP FloodingMachine learningIOTID-20, CICIDS-2017, BOT-IoT DatasetYesYesYes


## 3. The MidSiot IDS

### 3.1. System Overview

In this section, we explain how our proposal works. First of all, Figure1 illustrates the architecture of the proposed IDS that comprises three stages distributed between local network infrastructures and ISPs. The first stage is operated at the local gateways to identify connected IoT devices through their network behaviors. The next stage, which is conducted at the Internet gateways of ISPs, classifies network traffic of such IoT devices as normality or abnormality. When abnormal traffic is detected, it is transferred to the third stage to identify the attack types. Since the last two stages are done on ISPs which aggregate a huge volume of network traffic, correctly identifying the IoT device types along with their network traffic at the first stage is essential in increasing attack detection performance at following stages, especially for large-scale attacks targeting at multiple networks.Figure 1
Overview of the architecture of the multistage intrusion detection system.Second, Figures2 and 3 present the block diagram of main operational phases in MidSiot, including the training and prediction phases, respectively. They also illustrate the connection and interfaces of components of the proposed IDS. In more detail, as shown in Figure 2, the raw network packets are captured by the packet flow inspection component from network traffic and transformed into network flows. These flows are then fed into the feature extraction component, extracting network features and computing network flow statistics. In addition, feature selection algorithm is applied to filter inappropriate features from the output features. In the training phase, these features are aggregated into a dataset used to train the models. Once models are trained successfully, the ISPs store these models used for the second and third stages in their local storages, while the models used for the first stage are sent to the local gateways. We note that the second stage employs several models, and each model is responsible for classifying network traffic for a specific device type.Figure 2
MidSiot’s training phase.Figure 3
MidSiot’s prediction phase.In the prediction phase illustrated in Figure3, the network features are constructed similarly with the training phase; however, they are then fed to models for detecting malicious traffic. In more detail, the model of the first stage running on the local gateway is loaded to identify the device type of such features. All this information is transferred to ISP’s Internet gateways, where a well-trained model corresponding to the device type is used to detect abnormality in these features. If malicious activities are detected, they are forwarded to the third stage to detect attack types by using a universal attack detection model. The detection results, including the IoT device under the attack and the type of attack, are sent to the action manager component to trigger necessary actions (logging attack behaviors, blocking the network traffic of victims, notifying administrators about the attack). Note that, because MidSiot’s structure uses linked stages, the errors of one stage might affect not only the following stages but also the overall system’s performance. For example, if MidSiot misclassifies the device type, the second-stage results are potentially false. This is because the second-stage model is trained to learn the network patterns associated with a specific device type, and these patterns are different for each device type. Similarly, if the second-stage model misclassifies normal network traffic as abnormal, the final stage result is incorrect and triggers a false alert.


### 3.2. Network Flow Generation

Network flow generator is used to generate network flows from a batch of raw network packets. In MidSiot, it is powered by the deep packet inspection method that aggregates packets into flows sharing source/destination IP, source/destination port, and protocol and calculates flow features and statistics. In addition, this method supports extracting MAC addresses, making it possible to label devices. Therefore, we could obtain 83 network features (e.g., FlowID, SourceIP, DestinationIP, SourcePort, DestinationPort, TimeStamp, and Protocols) listed in Appendix VI (Table2).Table 2
Extract network features.
Feature nameDescriptionfl_durFlow durationtot_fw_pkTotal packets in the forward directiontot_bw_pkTotal packets in the backward directiontot_l_fw_pktTotal size of the packet in the forward directionfw_pkt_l_maxMaximum size of the packet in the forward directionfw_pkt_l_minMinimum size of the packet in the forward directionfw_pkt_l_avgAverage size of the packet in the forward directionfw_pkt_l_stdStandard deviation size of the packet in the forward directionBw_pkt_l_maxMaximum size of the packet in the backward directionBw_pkt_l_minMinimum size of the packet in the backward directionBw_pkt_l_avgMean size of the packet in the backward directionBw_pkt_l_stdStandard deviation size of the packet in the backward directionfl_byt_sFlow byte rate that is the number of packets transferred per secondfl_pkt_sFlow packets rate that is the number of packets transferred per secondfl_iat_avgAverage time between two flowsfl_iat_stdStandard deviation time two flowsfl_iat_maxMaximum time between two flowsfl_iat_minMinimum time between two flowsfw_iat_totTotal time between two packets sent in the forward directionfw_iat_avgMean time between two packets sent in the forward directionfw_iat_stdStandard deviation time between two packets sent in the forward directionfw_iat_maxMaximum time between two packets sent in the forward directionfw_iat_minMinimum time between two packets sent in the forward directionbw_iat_totTotal time between two packets sent in the backward directionbw_iat_avgMean time between two packets sent in the backward directionbw_iat_stdStandard deviation time between two packets sent in the backward directionbw_iat_maxMaximum time between two packets sent in the backward directionbw_iat_minMinimum time between two packets sent in the backward directionfw_psh_flagNumber of times the PSH flag was set in packets travelling in the forward direction (0 for UDP)bw_psh_flagNumber of times the PSH flag was set in packets travelling in the backward direction (0 for UDP)fw_urg_flagNumber of times the URG flag was set in packets travelling in the forward direction (0 for UDP)bw_urg_flagNumber of times the URG flag was set in packets travelling in the backward direction (0 for UDP)fw_hdr_lenTotal bytes used for headers in the forward directionbw_hdr_lenTotal bytes used for headers in the forward directionfw_pkt_sNumber of forward packets per secondbw_pkt_sNumber of backward packets per secondpkt_len_minMinimum length of a flowpkt_len_maxMaximum length of a flowpkt_len_avgMean length of a flowpkt_len_stdStandard deviation length of a flowpkt_len_vaMinimum interarrival time of the packetfin_cntNumber of packets with FINsyn_cntNumber of packets with SYNrst_cntNumber of packets with RSTpst_cntNumber of packets with PUSHack_cntNumber of packets with ACKurg_cntNumber of packets with URGcwe_cntNumber of packets with CWEece_cntNumber of packets with ECEdown_up_ratioDownload and upload ratiopkt_size_avgAverage size of packetfw_seg_avgAverage size observed in the forward directionbw_seg_avgAverage size observed in the backward directionfw_byt_blk_avgAverage number of bytes bulk rate in the forward directionfw_pkt_blk_avgAverage number of packets bulk rate in the forward directionfw_blk_rate_avgAverage number of bulk rate in the forward directionbw_byt_blk_avgAverage number of bytes bulk rate in the backward directionbw_pkt_blk_avgAverage number of packets bulk rate in the backward directionbw_blk_rate_avgAverage number of bulk rate in the backward directionsubfl_fw_pkThe average number of packets in a subflow in the forward directionsubfl_fw_bytThe average number of bytes in a subflow in the forward directionsubfl_bw_pktThe average number of packets in a subflow in the backward directionsubfl_bw_bytThe average number of bytes in a subflow in the backward directionfw_win_bytNumber of bytes sent in initial window in the forward directionbw_win_byt# of bytes sent in initial window in the backward directionFw_act_pkt# of packets with at least 1 byte of TCP data payload in the forward directionfw_seg_minMinimum segment size observed in the forward directionatv_avgMean time a flow was active before becoming idleatv_stdStandard deviation time a flow was active before becoming idleatv_maxMaximum time a flow was active before becoming idleatv_minMinimum time a flow was active before becoming idleidl_avgMean time a flow was idle before becoming activeidl_stdStandard deviation time a flow was idle before becoming activeidl_maxMaximum time a flow was idle before becoming activeidl_minMinimum time a flow was idle before becoming active


### 3.3. Data Preprocessing

In IDS datasets, not all features are suitable for machine learning algorithms; some of them may degrade the model training performance, whereas others make models overfit. Therefore, employing a feature selection algorithm is necessary. First, all identity-based features (e.g.,ip_src, ip_dst, flow_id, timestamp) are dropped to prevent the overfit issues, even features related to MAC addresses after labeling connected devices. We then adopt Pearson’s correlation coefficient to identify and remove unimportant features. In more detail, the importance index of each feature is its linear correlation coefficient value varying between −1 and 1. Finally, we remove all features having an importance index lower than 0.8. As a result, the final dataset only comprises 40 features, excluding all labels. We note that In MidSiot, Pearson correlation was only used during the training phase to construct a set of concise and suitable features for machine learning models. This feature set is then saved and loaded to the IDS in the detection phases. This means that Pearson correlation is inactive in the detection phase. Therefore, it has no impact on the detection procedure.Algorithm 1: Data preprocessing in training phases.
Input: Raw network packets XOutput: Processed data D(1)
Initialize:D=(2)
Generate network flows:F←NetworkFlowGeneratorX(3)
Label device type:LabelDeviceTypeF(4)
Drop invalid flows and identity columns:F1←DropInvalidDataF(5)
Normalize data:F2←NormalizeF1(6)
Select features using Pearson’s Correlation Coefficient:(a)
Fts←CorrF2(b)
D←SelectFeaturesF2,Fts(7)
ReturnDAlgorithm 1 Overview. Let X=x1,x2,…,xn denote a list of raw packets, and D is the list of processed flows. The major steps of this algorithm are described as follows:(1)
Network Flow Generation, in Line 2, receives a list of raw packets X and generates network flows by aggregating packets sharing Source/DestinationIP, Source/DestinationPort, Protocol.(2)
Label Device Types, in Line 3, the device type of each flow is deduced via Source/DestinationMACAddress. In addition, this step is performed during the training phase only.(3)
Drop and Normalize Data, in Lines 4 and 5, any flows in F having a null value at any field will be dropped. Afterward, the remaining flows are normalized to facilitate the machine learning processes.(4)
Pearson’s Correlation Coefficient, in Line 6, the Pearson’s correlation coefficient is applied on the normalized flows F2 to select only important features Fts. Finally, the flow list F2 will have some features dropped and only features from Ft are retained, which results in D.


### 3.4. Multistage Attack Detection Algorithm

#### 3.4.1. The Overview

The details of the multistage attack detection algorithm is described in Algorithm2. In more detail, let X=x1,x2,…,xn denote a list of raw packets and R is the resulting attack type. The entire detection process consists of the following main steps:(i)
Processing data, including generating network flows, dropping unnecessary features, and normalizing data are performed similarly to the Algorithm 1. However, as this is the detection process, device-type labeling and features selecting using Pearson’s correlation coefficient are inactive.(ii)
Classifying device type, the classification model from the storage of the local home gateway to perform prediction on the processed flow f2 to deduce the device type.(iii)
Sending the result to the ISP, the processed flow f2 in addition to the prediction results dt is forwarded to the ISP to further perform abnormality and attack-type detection.(iv)
Detecting the attacks, the model m is applied on the flow f2 to deduce whether that flow is normal or abnormal. If it is abnormal, move on to the next step; otherwise, mark this flow as null (which represents benign).(v)
Classifying attack type, a universal attack-type detection model is loaded from the storage of the ISP gateway. This model will be then applied on the malicious flow f2 to deduce the kind of attack a that has happened. At the end of this process, we know the device type of the flow, whether the flow is malicious or benign, and the attack type of the flow if it is malicious.Algorithm 2: The overall detection process.
Input: Raw network packets XOutput: Attack type R(i)
Local home gateway:(1)
Generate network flows:f←GenerateFlowsX(2)
Drop unnecessary features:f1←DropFeaturesf(3)
Normalize data:f2←Normalizef1(4)
Load device type classification model:M1←LoadModel(5)
Classify device type: Device typedt←M1.predictf2(6)
Send to ISP:Senddt,f2(ii)Internet Service Provider (ISP)(1)
Initialize:R=(2)
Load all abnormality detection models:M2←LoadModel2(3)
Load abnormality detection model:m←M2dt(4)
Attack detection:y←m.predictf2(5)
Ifyis notnormalthen(a)
Load attack type detection model:M3←LoadModel3(b)
Attack classification:a←M3.predictf2(c)
R←a(6)
ElseR←null(7)
ReturnR


#### 3.4.2. The First Stage

The primary benefit of the first stage is to enhance the accuracy of the attack detection model in the next steps. In more detail, since the IoT device types are various and heterogeneous, their network behaviors are highly diverse. For example, the high UDP packet rate coming from IoT cameras is normal, but the one from temperature sensors is a sign of a security threat. This may lead to false attack detection. Thus, identifying device types and considering them as an input feature of the attack detection model is crucial to increase the accuracy. Furthermore, this stage should be done on the local gateway for two reasons: (1) local gateways have sufficient computation power to handle a part of the detection process, which reduces the burden for the cloud; and (2) if device-type classification is done with the other two steps on the cloud, merge a multitude of network packets coming from various IoT networks. This aggregation may make the network data exhibit more generic characteristics than device-specific ones, reducing the device-type classification performance. This directly affects the attack detection accuracy. Therefore, running the device-type classifiers in the local gateways closed to IoT devices could mitigate this issue since only a limited number of device types are considered.


#### 3.4.3. The Second and Third Stages

Take a dataset(1)D=x1,y1,x2,y2,…,xN,yN,where xi=xi1,xi2,…,xinT, i=1,2,…,N, xi is the input instance that represents a network flow. xi has n features. N indicates the number of features of a network flow contained in the dataset D. yi∈0,1,2,…,K−1 is the result of each detection record. A decision tree recursively partitions the feature space such that the samples with the same labels or similar target values are grouped together. Let the data at node m be represented by Qm. For each split θ=j,tm consisting of a feature j and a threshold tm, partition the data into Qmleftθ and Qmrightθ subsets:(2)Qmleftθ=x,y|xi<=tm,Qmrightθ=Qm\Qmleftθ.The quality of a candidate split of nodem is then computed using an impurity function or loss function H, the choice of which depends on the task being solved (classification or regression):(3)GQm,θ=NmleftNmHQmleftθ+NmrightNmHQmrightθ.Select the parameters to minimize the impurity:(4)θ∗=argminθGQm,θ.Repeat for subsetsQmleftθ and Qmrightθ until the maximum allowable depth is reached Nm<minsample or Nm=1.For the classification of IDS,yi∈0,1,2,…,K−1 for node m represents a region of Rm with instances of Nm. Assume that pmk is the proportion of class k instance in m and can be obtained by the following formula:(5)pmk=1Nm∑xj∈RmIyi=k.The common measure of impurity is named Gini and can be obtained by the following formula:(6)HXm=−∑kpmk1−pmk.Cross-entropy can be obtained by the following formula:(7)HXm=−∑kpmklogpmk.Misclassification can be obtained by the following formula (not being used in the proposed system):(8)HXm=−∑k1−maxpmk.


## 3.1. System Overview

In this section, we explain how our proposal works. First of all, Figure1 illustrates the architecture of the proposed IDS that comprises three stages distributed between local network infrastructures and ISPs. The first stage is operated at the local gateways to identify connected IoT devices through their network behaviors. The next stage, which is conducted at the Internet gateways of ISPs, classifies network traffic of such IoT devices as normality or abnormality. When abnormal traffic is detected, it is transferred to the third stage to identify the attack types. Since the last two stages are done on ISPs which aggregate a huge volume of network traffic, correctly identifying the IoT device types along with their network traffic at the first stage is essential in increasing attack detection performance at following stages, especially for large-scale attacks targeting at multiple networks.Figure 1
Overview of the architecture of the multistage intrusion detection system.Second, Figures2 and 3 present the block diagram of main operational phases in MidSiot, including the training and prediction phases, respectively. They also illustrate the connection and interfaces of components of the proposed IDS. In more detail, as shown in Figure 2, the raw network packets are captured by the packet flow inspection component from network traffic and transformed into network flows. These flows are then fed into the feature extraction component, extracting network features and computing network flow statistics. In addition, feature selection algorithm is applied to filter inappropriate features from the output features. In the training phase, these features are aggregated into a dataset used to train the models. Once models are trained successfully, the ISPs store these models used for the second and third stages in their local storages, while the models used for the first stage are sent to the local gateways. We note that the second stage employs several models, and each model is responsible for classifying network traffic for a specific device type.Figure 2
MidSiot’s training phase.Figure 3
MidSiot’s prediction phase.In the prediction phase illustrated in Figure3, the network features are constructed similarly with the training phase; however, they are then fed to models for detecting malicious traffic. In more detail, the model of the first stage running on the local gateway is loaded to identify the device type of such features. All this information is transferred to ISP’s Internet gateways, where a well-trained model corresponding to the device type is used to detect abnormality in these features. If malicious activities are detected, they are forwarded to the third stage to detect attack types by using a universal attack detection model. The detection results, including the IoT device under the attack and the type of attack, are sent to the action manager component to trigger necessary actions (logging attack behaviors, blocking the network traffic of victims, notifying administrators about the attack). Note that, because MidSiot’s structure uses linked stages, the errors of one stage might affect not only the following stages but also the overall system’s performance. For example, if MidSiot misclassifies the device type, the second-stage results are potentially false. This is because the second-stage model is trained to learn the network patterns associated with a specific device type, and these patterns are different for each device type. Similarly, if the second-stage model misclassifies normal network traffic as abnormal, the final stage result is incorrect and triggers a false alert.


## 3.2. Network Flow Generation

Network flow generator is used to generate network flows from a batch of raw network packets. In MidSiot, it is powered by the deep packet inspection method that aggregates packets into flows sharing source/destination IP, source/destination port, and protocol and calculates flow features and statistics. In addition, this method supports extracting MAC addresses, making it possible to label devices. Therefore, we could obtain 83 network features (e.g., FlowID, SourceIP, DestinationIP, SourcePort, DestinationPort, TimeStamp, and Protocols) listed in Appendix VI (Table2).Table 2
Extract network features.
Feature nameDescriptionfl_durFlow durationtot_fw_pkTotal packets in the forward directiontot_bw_pkTotal packets in the backward directiontot_l_fw_pktTotal size of the packet in the forward directionfw_pkt_l_maxMaximum size of the packet in the forward directionfw_pkt_l_minMinimum size of the packet in the forward directionfw_pkt_l_avgAverage size of the packet in the forward directionfw_pkt_l_stdStandard deviation size of the packet in the forward directionBw_pkt_l_maxMaximum size of the packet in the backward directionBw_pkt_l_minMinimum size of the packet in the backward directionBw_pkt_l_avgMean size of the packet in the backward directionBw_pkt_l_stdStandard deviation size of the packet in the backward directionfl_byt_sFlow byte rate that is the number of packets transferred per secondfl_pkt_sFlow packets rate that is the number of packets transferred per secondfl_iat_avgAverage time between two flowsfl_iat_stdStandard deviation time two flowsfl_iat_maxMaximum time between two flowsfl_iat_minMinimum time between two flowsfw_iat_totTotal time between two packets sent in the forward directionfw_iat_avgMean time between two packets sent in the forward directionfw_iat_stdStandard deviation time between two packets sent in the forward directionfw_iat_maxMaximum time between two packets sent in the forward directionfw_iat_minMinimum time between two packets sent in the forward directionbw_iat_totTotal time between two packets sent in the backward directionbw_iat_avgMean time between two packets sent in the backward directionbw_iat_stdStandard deviation time between two packets sent in the backward directionbw_iat_maxMaximum time between two packets sent in the backward directionbw_iat_minMinimum time between two packets sent in the backward directionfw_psh_flagNumber of times the PSH flag was set in packets travelling in the forward direction (0 for UDP)bw_psh_flagNumber of times the PSH flag was set in packets travelling in the backward direction (0 for UDP)fw_urg_flagNumber of times the URG flag was set in packets travelling in the forward direction (0 for UDP)bw_urg_flagNumber of times the URG flag was set in packets travelling in the backward direction (0 for UDP)fw_hdr_lenTotal bytes used for headers in the forward directionbw_hdr_lenTotal bytes used for headers in the forward directionfw_pkt_sNumber of forward packets per secondbw_pkt_sNumber of backward packets per secondpkt_len_minMinimum length of a flowpkt_len_maxMaximum length of a flowpkt_len_avgMean length of a flowpkt_len_stdStandard deviation length of a flowpkt_len_vaMinimum interarrival time of the packetfin_cntNumber of packets with FINsyn_cntNumber of packets with SYNrst_cntNumber of packets with RSTpst_cntNumber of packets with PUSHack_cntNumber of packets with ACKurg_cntNumber of packets with URGcwe_cntNumber of packets with CWEece_cntNumber of packets with ECEdown_up_ratioDownload and upload ratiopkt_size_avgAverage size of packetfw_seg_avgAverage size observed in the forward directionbw_seg_avgAverage size observed in the backward directionfw_byt_blk_avgAverage number of bytes bulk rate in the forward directionfw_pkt_blk_avgAverage number of packets bulk rate in the forward directionfw_blk_rate_avgAverage number of bulk rate in the forward directionbw_byt_blk_avgAverage number of bytes bulk rate in the backward directionbw_pkt_blk_avgAverage number of packets bulk rate in the backward directionbw_blk_rate_avgAverage number of bulk rate in the backward directionsubfl_fw_pkThe average number of packets in a subflow in the forward directionsubfl_fw_bytThe average number of bytes in a subflow in the forward directionsubfl_bw_pktThe average number of packets in a subflow in the backward directionsubfl_bw_bytThe average number of bytes in a subflow in the backward directionfw_win_bytNumber of bytes sent in initial window in the forward directionbw_win_byt# of bytes sent in initial window in the backward directionFw_act_pkt# of packets with at least 1 byte of TCP data payload in the forward directionfw_seg_minMinimum segment size observed in the forward directionatv_avgMean time a flow was active before becoming idleatv_stdStandard deviation time a flow was active before becoming idleatv_maxMaximum time a flow was active before becoming idleatv_minMinimum time a flow was active before becoming idleidl_avgMean time a flow was idle before becoming activeidl_stdStandard deviation time a flow was idle before becoming activeidl_maxMaximum time a flow was idle before becoming activeidl_minMinimum time a flow was idle before becoming active


## 3.3. Data Preprocessing

In IDS datasets, not all features are suitable for machine learning algorithms; some of them may degrade the model training performance, whereas others make models overfit. Therefore, employing a feature selection algorithm is necessary. First, all identity-based features (e.g.,ip_src, ip_dst, flow_id, timestamp) are dropped to prevent the overfit issues, even features related to MAC addresses after labeling connected devices. We then adopt Pearson’s correlation coefficient to identify and remove unimportant features. In more detail, the importance index of each feature is its linear correlation coefficient value varying between −1 and 1. Finally, we remove all features having an importance index lower than 0.8. As a result, the final dataset only comprises 40 features, excluding all labels. We note that In MidSiot, Pearson correlation was only used during the training phase to construct a set of concise and suitable features for machine learning models. This feature set is then saved and loaded to the IDS in the detection phases. This means that Pearson correlation is inactive in the detection phase. Therefore, it has no impact on the detection procedure.Algorithm 1: Data preprocessing in training phases.
Input: Raw network packets XOutput: Processed data D(1)
Initialize:D=(2)
Generate network flows:F←NetworkFlowGeneratorX(3)
Label device type:LabelDeviceTypeF(4)
Drop invalid flows and identity columns:F1←DropInvalidDataF(5)
Normalize data:F2←NormalizeF1(6)
Select features using Pearson’s Correlation Coefficient:(a)
Fts←CorrF2(b)
D←SelectFeaturesF2,Fts(7)
ReturnDAlgorithm 1 Overview. Let X=x1,x2,…,xn denote a list of raw packets, and D is the list of processed flows. The major steps of this algorithm are described as follows:(1)
Network Flow Generation, in Line 2, receives a list of raw packets X and generates network flows by aggregating packets sharing Source/DestinationIP, Source/DestinationPort, Protocol.(2)
Label Device Types, in Line 3, the device type of each flow is deduced via Source/DestinationMACAddress. In addition, this step is performed during the training phase only.(3)
Drop and Normalize Data, in Lines 4 and 5, any flows in F having a null value at any field will be dropped. Afterward, the remaining flows are normalized to facilitate the machine learning processes.(4)
Pearson’s Correlation Coefficient, in Line 6, the Pearson’s correlation coefficient is applied on the normalized flows F2 to select only important features Fts. Finally, the flow list F2 will have some features dropped and only features from Ft are retained, which results in D.


## 3.4. Multistage Attack Detection Algorithm

### 3.4.1. The Overview

The details of the multistage attack detection algorithm is described in Algorithm2. In more detail, let X=x1,x2,…,xn denote a list of raw packets and R is the resulting attack type. The entire detection process consists of the following main steps:(i)
Processing data, including generating network flows, dropping unnecessary features, and normalizing data are performed similarly to the Algorithm 1. However, as this is the detection process, device-type labeling and features selecting using Pearson’s correlation coefficient are inactive.(ii)
Classifying device type, the classification model from the storage of the local home gateway to perform prediction on the processed flow f2 to deduce the device type.(iii)
Sending the result to the ISP, the processed flow f2 in addition to the prediction results dt is forwarded to the ISP to further perform abnormality and attack-type detection.(iv)
Detecting the attacks, the model m is applied on the flow f2 to deduce whether that flow is normal or abnormal. If it is abnormal, move on to the next step; otherwise, mark this flow as null (which represents benign).(v)
Classifying attack type, a universal attack-type detection model is loaded from the storage of the ISP gateway. This model will be then applied on the malicious flow f2 to deduce the kind of attack a that has happened. At the end of this process, we know the device type of the flow, whether the flow is malicious or benign, and the attack type of the flow if it is malicious.Algorithm 2: The overall detection process.
Input: Raw network packets XOutput: Attack type R(i)
Local home gateway:(1)
Generate network flows:f←GenerateFlowsX(2)
Drop unnecessary features:f1←DropFeaturesf(3)
Normalize data:f2←Normalizef1(4)
Load device type classification model:M1←LoadModel(5)
Classify device type: Device typedt←M1.predictf2(6)
Send to ISP:Senddt,f2(ii)Internet Service Provider (ISP)(1)
Initialize:R=(2)
Load all abnormality detection models:M2←LoadModel2(3)
Load abnormality detection model:m←M2dt(4)
Attack detection:y←m.predictf2(5)
Ifyis notnormalthen(a)
Load attack type detection model:M3←LoadModel3(b)
Attack classification:a←M3.predictf2(c)
R←a(6)
ElseR←null(7)
ReturnR


### 3.4.2. The First Stage

The primary benefit of the first stage is to enhance the accuracy of the attack detection model in the next steps. In more detail, since the IoT device types are various and heterogeneous, their network behaviors are highly diverse. For example, the high UDP packet rate coming from IoT cameras is normal, but the one from temperature sensors is a sign of a security threat. This may lead to false attack detection. Thus, identifying device types and considering them as an input feature of the attack detection model is crucial to increase the accuracy. Furthermore, this stage should be done on the local gateway for two reasons: (1) local gateways have sufficient computation power to handle a part of the detection process, which reduces the burden for the cloud; and (2) if device-type classification is done with the other two steps on the cloud, merge a multitude of network packets coming from various IoT networks. This aggregation may make the network data exhibit more generic characteristics than device-specific ones, reducing the device-type classification performance. This directly affects the attack detection accuracy. Therefore, running the device-type classifiers in the local gateways closed to IoT devices could mitigate this issue since only a limited number of device types are considered.


### 3.4.3. The Second and Third Stages

Take a dataset(1)D=x1,y1,x2,y2,…,xN,yN,where xi=xi1,xi2,…,xinT, i=1,2,…,N, xi is the input instance that represents a network flow. xi has n features. N indicates the number of features of a network flow contained in the dataset D. yi∈0,1,2,…,K−1 is the result of each detection record. A decision tree recursively partitions the feature space such that the samples with the same labels or similar target values are grouped together. Let the data at node m be represented by Qm. For each split θ=j,tm consisting of a feature j and a threshold tm, partition the data into Qmleftθ and Qmrightθ subsets:(2)Qmleftθ=x,y|xi<=tm,Qmrightθ=Qm\Qmleftθ.The quality of a candidate split of nodem is then computed using an impurity function or loss function H, the choice of which depends on the task being solved (classification or regression):(3)GQm,θ=NmleftNmHQmleftθ+NmrightNmHQmrightθ.Select the parameters to minimize the impurity:(4)θ∗=argminθGQm,θ.Repeat for subsetsQmleftθ and Qmrightθ until the maximum allowable depth is reached Nm<minsample or Nm=1.For the classification of IDS,yi∈0,1,2,…,K−1 for node m represents a region of Rm with instances of Nm. Assume that pmk is the proportion of class k instance in m and can be obtained by the following formula:(5)pmk=1Nm∑xj∈RmIyi=k.The common measure of impurity is named Gini and can be obtained by the following formula:(6)HXm=−∑kpmk1−pmk.Cross-entropy can be obtained by the following formula:(7)HXm=−∑kpmklogpmk.Misclassification can be obtained by the following formula (not being used in the proposed system):(8)HXm=−∑k1−maxpmk.


## 3.4.1. The Overview

The details of the multistage attack detection algorithm is described in Algorithm2. In more detail, let X=x1,x2,…,xn denote a list of raw packets and R is the resulting attack type. The entire detection process consists of the following main steps:(i)
Processing data, including generating network flows, dropping unnecessary features, and normalizing data are performed similarly to the Algorithm 1. However, as this is the detection process, device-type labeling and features selecting using Pearson’s correlation coefficient are inactive.(ii)
Classifying device type, the classification model from the storage of the local home gateway to perform prediction on the processed flow f2 to deduce the device type.(iii)
Sending the result to the ISP, the processed flow f2 in addition to the prediction results dt is forwarded to the ISP to further perform abnormality and attack-type detection.(iv)
Detecting the attacks, the model m is applied on the flow f2 to deduce whether that flow is normal or abnormal. If it is abnormal, move on to the next step; otherwise, mark this flow as null (which represents benign).(v)
Classifying attack type, a universal attack-type detection model is loaded from the storage of the ISP gateway. This model will be then applied on the malicious flow f2 to deduce the kind of attack a that has happened. At the end of this process, we know the device type of the flow, whether the flow is malicious or benign, and the attack type of the flow if it is malicious.Algorithm 2: The overall detection process.
Input: Raw network packets XOutput: Attack type R(i)
Local home gateway:(1)
Generate network flows:f←GenerateFlowsX(2)
Drop unnecessary features:f1←DropFeaturesf(3)
Normalize data:f2←Normalizef1(4)
Load device type classification model:M1←LoadModel(5)
Classify device type: Device typedt←M1.predictf2(6)
Send to ISP:Senddt,f2(ii)Internet Service Provider (ISP)(1)
Initialize:R=(2)
Load all abnormality detection models:M2←LoadModel2(3)
Load abnormality detection model:m←M2dt(4)
Attack detection:y←m.predictf2(5)
Ifyis notnormalthen(a)
Load attack type detection model:M3←LoadModel3(b)
Attack classification:a←M3.predictf2(c)
R←a(6)
ElseR←null(7)
ReturnR


## 3.4.2. The First Stage

The primary benefit of the first stage is to enhance the accuracy of the attack detection model in the next steps. In more detail, since the IoT device types are various and heterogeneous, their network behaviors are highly diverse. For example, the high UDP packet rate coming from IoT cameras is normal, but the one from temperature sensors is a sign of a security threat. This may lead to false attack detection. Thus, identifying device types and considering them as an input feature of the attack detection model is crucial to increase the accuracy. Furthermore, this stage should be done on the local gateway for two reasons: (1) local gateways have sufficient computation power to handle a part of the detection process, which reduces the burden for the cloud; and (2) if device-type classification is done with the other two steps on the cloud, merge a multitude of network packets coming from various IoT networks. This aggregation may make the network data exhibit more generic characteristics than device-specific ones, reducing the device-type classification performance. This directly affects the attack detection accuracy. Therefore, running the device-type classifiers in the local gateways closed to IoT devices could mitigate this issue since only a limited number of device types are considered.


## 3.4.3. The Second and Third Stages

Take a dataset(1)D=x1,y1,x2,y2,…,xN,yN,where xi=xi1,xi2,…,xinT, i=1,2,…,N, xi is the input instance that represents a network flow. xi has n features. N indicates the number of features of a network flow contained in the dataset D. yi∈0,1,2,…,K−1 is the result of each detection record. A decision tree recursively partitions the feature space such that the samples with the same labels or similar target values are grouped together. Let the data at node m be represented by Qm. For each split θ=j,tm consisting of a feature j and a threshold tm, partition the data into Qmleftθ and Qmrightθ subsets:(2)Qmleftθ=x,y|xi<=tm,Qmrightθ=Qm\Qmleftθ.The quality of a candidate split of nodem is then computed using an impurity function or loss function H, the choice of which depends on the task being solved (classification or regression):(3)GQm,θ=NmleftNmHQmleftθ+NmrightNmHQmrightθ.Select the parameters to minimize the impurity:(4)θ∗=argminθGQm,θ.Repeat for subsetsQmleftθ and Qmrightθ until the maximum allowable depth is reached Nm<minsample or Nm=1.For the classification of IDS,yi∈0,1,2,…,K−1 for node m represents a region of Rm with instances of Nm. Assume that pmk is the proportion of class k instance in m and can be obtained by the following formula:(5)pmk=1Nm∑xj∈RmIyi=k.The common measure of impurity is named Gini and can be obtained by the following formula:(6)HXm=−∑kpmk1−pmk.Cross-entropy can be obtained by the following formula:(7)HXm=−∑kpmklogpmk.Misclassification can be obtained by the following formula (not being used in the proposed system):(8)HXm=−∑k1−maxpmk.


## 4. Results and Discussion

### 4.1. Evaluation Metrics

In our experiments, we adopted several evaluation metrics, such as precision (P), recall (R), F-measure (F), and accuracy. Let TP, FP, and FN denote true positives, false positives, and false negatives, respectively, and these evaluation metrics are defined as(9)P=TPTP+FP,R=TPTP+FN,F=2.P.RP+R.To evaluate the quality of attack detection models, we use the model accuracy as a primary evaluation metric that is directly computed from the confusion matrix based on the following formula:(10)Accuracy=TP+TNTP+TN+FP+FN.We evaluate the performance of these resampling methods through macroaverageF1-score performed independently on each class. Let MAP and MAR denote macroaverage precision and macroaverage recall, and the macroaverage F1-score is defined as(11)MacroAvgF1=2∗MacroAvgPrec∗MacroAvgRecMacroAvgPrec−1+MacroAvgRec−1,where(12)MacroAvgPrec=∑k=1KPrecisionkK,MacroAvgRec=∑k=1KRecallkK,whereK is the total number of classes.


### 4.2. Dataset and Attack Class Balancing

We assessed MidSiot on three different datasets: IoTID20, CIC-IDS-2017, and BOT-IoT.(i)
The IoTID20 dataset consists of two IoT devices (a smart home device SKT NGU and an EZVIZ Wi-Fi camera) and several non-IoT devices marked as external devices. The cyberattacks on these devices are classified into four attack categories and seven attack subcategories described in detail in Table3.(ii)
The CIC-IDS-2017 dataset contains the network traffic of six cyberattack types listed in Table4 targeting 12 different IoT devices, which are labeled according to their operating systems and architectures. It also has several external devices to generate normal traffic.(iii)
The BOT-IoT has five IoT devices and several external devices. The malicious network traffic of these devices is classified into three attack categories and detailed in Table5.Table 3
Devices, attack categories, and subcategories of the IoTID20 dataset.
DeviceCategorySubcategoryEZVIZ,NUGU,ExternalNormal,DoS,Mirai, MITM,ScanNormal,Syn Flooding, Brute Force, HTTP Flooding, UDP Flooding,ARP Spoofing, Host Port, OSTable 4
Devices, attack categories, and subcategories of the CIC-IDS-2017 dataset.
DeviceCategorySubcategoryWeb server 16 Public,Benign,Benign,Ubuntu server 12 Public,Bot,Bot,Ubuntu 14.4 32bit,Brute Force,FTP-Patator,Ubuntu 14.4 64bit,Dos/Ddos,SSH-Patator,Ubuntu 16.4 32bit,Infiltration,DDoS, DoS, GoldenEye,Ubuntu 16.4 64bit,Portscan,DoS Hulk, DoS Slow,Win 7 Pro,Web Attackhttptest, DoS slowloris,Win 8.1 64bit,Infiltration, Portscan,Win Vista 64bit,Web Attack-Brute Force,Win 10 pro 32bit,Web Attack-Sql Injection,win 10 64bit,Web Attack-XSS, HeartbleedMAC,ExternalTable 5
Devices, attack categories, and subcategories of the BOT-IoT dataset.
DeviceCategorySubcategoryUbuntu Server,Normal,Normal,Ubuntu Mobile,DoS/DDoS,Service scanning,Metasploitable,Reconnaissance,OS Fingerprinting,Windows 7,TheftDoS/DDoS TCP, DoS/DDoS UDP,Ubuntu Tap,DoS/DDoS HTTP,ExternalKeylogging, data exfiltrationThrough rigorously analyzing the evaluation datasets, we figured out that the number of samples of each attack type is slightly imbalanced; thus, employing resampling methods is necessary. In our experiments, we experimented and evaluated the Random Undersampling algorithm (RU) and its conjunction with Synthetic Minority Oversampling Technique (RU-SMOTE).(i)
Random Undersampling is a random selection process running on overwhelmed attack types to reduce their size. However, randomly selecting data points might accidently ignore critical information, resulting in degraded classification performance.(ii)
Synthetic Minority Oversampling Technique (SMOTE) balances the dataset by synthesizing new samples for the minority class. In more detail, it selects a cluster of samples and draws a line between them; new samples are the points along this line.To implement these resampling methods, we utilizedimblearn library [33] supporting multiple resampling techniques along with several running strategies for both binary and multiclass classification. In binary classification, we need to configure the ratio between minority class and majority class after resampling, whereas this configuration is unnecessary in multiclass classification. From the results illustrated in Figures 4-6, it is obvious that these resampling techniques have no significant impact on the performance of models. Furthermore, if the number of samples in majority classes drastically outweighs ones of minority classes, they may decrease the attack detection quality of the smaller classes. Therefore, our models are trained without resampling techniques.Figure 4
The macroaverageF1-score of the IoTID20 dataset.Figure 5
The macroaverageF1-score of the CIC-IDS-2017 dataset.Figure 6
The macroaverageF1-score of the BOT-IoT dataset.


### 4.3. Results and Discussion

To select classification algorithms for our multistage IDS, we examined the detection time and accuracy of several supervised machine learning algorithms on the IoTID20 dataset. In detail, the ability to classify benign and malicious network traffic (binary attack detection) and identify precisely the types of attacks (multiclass attack detection) are both considered. Since operating on network gateways requires a lightweight attack detection model, experimented algorithms are simple machine learning algorithms, including linear support vector machine, quadratic support vector machine, K-nearest-neighbor, linear discriminant analysis, quadratic discriminant analysis, multilayer perceptron, long short-term memory, autoencoder classifier, and decision tree classifier; their results are presented in Table6. As shown in the table, the decision tree classifier outperforms other algorithms, and it is considered a lightweight machine learning algorithm [34]. This classifier is thus selected for the attack detection model of MidSiot. Note that empty values in the table (denoted by N/A) imply the long training time exceeding two hours. Moreover, training a decision tree classifier is trivial and possibly performed on IoT devices. We also applied Classification and Regression Tree (CART) to boost the detection performance. In more detail, CART splits training data into two subsets based on a specific feature k and a threshold tk (e.g., “flow duration ≤ 100”). This split is repeated on each subset until it reaches the maximum depth or subset size equals to 0. As a result, the computational complexity of the classifier is reduced to Olog2m with m being the number of samples in the training set. This significantly increases the training and prediction rate to deal with large datasets.Table 6
The results of examined classifiers.
ModelClassificationAccuracy (%)Training timeμs/flowPrediction timeμs/flowAdjusted parametersLinear support vector machineBinary98.161150.57204.54kernel = lineargamma = autoMulticlassN/A10 298.31453.1Quadratic support vector machineBinary98.25792.61311.5kernel = polygamma = autoMulticlassN/AN/AN/AK-Nearest neighborBinary99.790.172343.72n_neighbors = 5Multiclass98.610.192377.81Linear discriminant analysisBinary95.0721.60.27All defaultMulticlass80.7325.714.35Quadratic discriminant analysisBinary53.618.8912.44All defaultMulticlass56.6216.6214.74Multilayer perceptronBinary99.62.570.65Input layer and first layer with 50 neurons and activation = reluOutput layer with activation = sigmoidMulticlass92.714.947.96Long short-term memoryBinary96.53572.4462.64input layer and LSTM layer with 50 neuronsOutput layer with activation = sigmoidMulticlassN/AN/AN/AAutoencoder classifierBinary93.0111.650.62Encoding layer with 50 neurons and activation = reluDecoding and output layer with activation = softmaxMulticlass87.7413.350.81Decision tree classifierBinary99.9412.490.43criterion = entropyMulticlass99.6916.760.38Attack detection results: Table 7 reports the overall performance of each MidSiot’s stage on evaluated datasets. We can see that MidSiot could not only accurately differentiate between normal and malicious traffic in the second stage, but also identify the type of attacks in the third stage. The average accuracy of such stages is about 99.98% and 99.68%, respectively. Regarding classifying IoT devices, our proposal achieved a high classification accuracy reported at 95.55% on average. In detail, device-type classification for BOT-IoT achieves the highest result, at 99.92%, whereas the results of IoTID20 and CIC-IDS-2017 are about 92.57% and 94.17%, respectively. To have a better understanding the attack detection performance of MidSiot, Tables 8–10 illustrate the confusion matrices, which present the comparison between predicted attacks and the actual ones.Table 7
The evaluation results of the multistage IDS.
DatasetStageAccuracy%Training timeμs/flowPrediction timeμs/flowIoTID20192.5735.760.44299.9511.330.26399.1515.90.31CIC-IDS-2017194.17110.310.89299.9925.650.25399.9778.680.27BOT-IoT199.9219.270.15299.993.240.07399.9315.050.17Table 8
Attack detection confusion matrix on the IoTID20 dataset.
PredictedMiraiScanDoSNormalMITM ARP spoofingActualMirai4124462583038569Scan61874 47309155DoS925935653Normal631565330MITM ARP Spoofing62420021634509Table 9
Attack detection confusion matrix on the CIC-IDS-2017 dataset.
PredictedBenignDoS/DDoSPortscanBotWeb attackActualBenign15830362735DoS/DDoS19235881802Portscan6615891101Bot100019290Web Attack401201463Table 10
Attack detection confusion matrix on the BOT-IoT dataset.
PredictedDDoSDoSReconnaissanceNormalActualDDoS192618943500DoS131983078600Reconnaissance26818182Normal1132Comparing with baseline methods: We compared attack detection quality between our proposal and state-of-the-art IDS and reported the results in Table 11. Overall, MidSiot outperforms its competitors on CICIDS-2017 and BOT-IoT datasets and is comparable with them on IoTID2020 dataset. In more detail, regarding CIC-IDS-2017, the best of our competitors achieves 99.9% in both binary and multiclass classification, whereas our proposed IDS achieves better results recorded about 99.99% and 99.97%, respectively. Similar results are also found in the BOT-IoT dataset, in which our proposal achieves 99.99% accuracy in binary classification and 99.93% in multiclass classification problems. In the IoTID20 dataset, the best competitor detects the attack types with 100% accuracy, and MidSiot also has very competitive results reported at 99.15%. Compared with state-of-the-art IDSs, MidSiot employed more machine learning models to enhance detection accuracy. This demands high computation costs and training datasets to train these models. Indeed, each stage in MidSiot has a different training dataset, which requires a huge effort to label. For example, the first-stage model needs to label the device type of network traffic, whereas the second-stage model demands labeling abnormal traffic. Moreover, deploying the first-stage model from the cloud to IoT local gateway consumes network bandwidth and may trigger delays.Table 11
Comparing the proposed system with other one-stage systems.
DatasetWorkAttack detection modelBinary attack detection%Multiclass attack detection%Prediction timeμs/netflowCICIDS-2017MidSiotMultistage99.9999.971.41Gamage and Samarabandu [35]Random forestN/A99.8660.48Vinayakumar et al. [36]Deep neural network93.1095.60N/AElmrabit et al. [37]Decision tree99.9099.90N/AManimuruganDeep belief network99.3797.73N/AIOTID20MidSiotMultistage99.9899.881.01Ullah and Mahmoud [21]Decision tree99.9499.69N/AAlkahtani and Aldhyani [38]Long short-term memory98.20N/AN/ASong et al. [39]Autoencoder93.7695.20N/AHussein et al. [40]Random forest99.9099.90N/AUllah and Mahmoud [41]Convolution neural network99.9897.76N/AIslam et al. [42]Decision treeN/A100.001139.37BOT-IOTMidSiotMultistage99.9999.990.39Ferrag et al. [43]Rules and decision treeN/A97.001.54Ferrag et al. [44]Deep autoencoderN/A98.391916.55Dwibedi et al. [45]Support vector machine99.99N/AN/APokhrel et al. [46]K-Nearest neighbor92.10N/AN/AGe et al. [47]Support vector machine99.7499.03693 040Ullah and Mahmoud [41]Convolution neural network99.9099.97N/AIn conclusion, by using a hierarchical architecture and chaining stages together, MidSiot effectively classifies device types, identifies abnormal network traffic, and differentiates cyberattack types.


## 4.1. Evaluation Metrics

In our experiments, we adopted several evaluation metrics, such as precision (P), recall (R), F-measure (F), and accuracy. Let TP, FP, and FN denote true positives, false positives, and false negatives, respectively, and these evaluation metrics are defined as(9)P=TPTP+FP,R=TPTP+FN,F=2.P.RP+R.To evaluate the quality of attack detection models, we use the model accuracy as a primary evaluation metric that is directly computed from the confusion matrix based on the following formula:(10)Accuracy=TP+TNTP+TN+FP+FN.We evaluate the performance of these resampling methods through macroaverageF1-score performed independently on each class. Let MAP and MAR denote macroaverage precision and macroaverage recall, and the macroaverage F1-score is defined as(11)MacroAvgF1=2∗MacroAvgPrec∗MacroAvgRecMacroAvgPrec−1+MacroAvgRec−1,where(12)MacroAvgPrec=∑k=1KPrecisionkK,MacroAvgRec=∑k=1KRecallkK,whereK is the total number of classes.


## 4.2. Dataset and Attack Class Balancing

We assessed MidSiot on three different datasets: IoTID20, CIC-IDS-2017, and BOT-IoT.(i)
The IoTID20 dataset consists of two IoT devices (a smart home device SKT NGU and an EZVIZ Wi-Fi camera) and several non-IoT devices marked as external devices. The cyberattacks on these devices are classified into four attack categories and seven attack subcategories described in detail in Table3.(ii)
The CIC-IDS-2017 dataset contains the network traffic of six cyberattack types listed in Table4 targeting 12 different IoT devices, which are labeled according to their operating systems and architectures. It also has several external devices to generate normal traffic.(iii)
The BOT-IoT has five IoT devices and several external devices. The malicious network traffic of these devices is classified into three attack categories and detailed in Table5.Table 3
Devices, attack categories, and subcategories of the IoTID20 dataset.
DeviceCategorySubcategoryEZVIZ,NUGU,ExternalNormal,DoS,Mirai, MITM,ScanNormal,Syn Flooding, Brute Force, HTTP Flooding, UDP Flooding,ARP Spoofing, Host Port, OSTable 4
Devices, attack categories, and subcategories of the CIC-IDS-2017 dataset.
DeviceCategorySubcategoryWeb server 16 Public,Benign,Benign,Ubuntu server 12 Public,Bot,Bot,Ubuntu 14.4 32bit,Brute Force,FTP-Patator,Ubuntu 14.4 64bit,Dos/Ddos,SSH-Patator,Ubuntu 16.4 32bit,Infiltration,DDoS, DoS, GoldenEye,Ubuntu 16.4 64bit,Portscan,DoS Hulk, DoS Slow,Win 7 Pro,Web Attackhttptest, DoS slowloris,Win 8.1 64bit,Infiltration, Portscan,Win Vista 64bit,Web Attack-Brute Force,Win 10 pro 32bit,Web Attack-Sql Injection,win 10 64bit,Web Attack-XSS, HeartbleedMAC,ExternalTable 5
Devices, attack categories, and subcategories of the BOT-IoT dataset.
DeviceCategorySubcategoryUbuntu Server,Normal,Normal,Ubuntu Mobile,DoS/DDoS,Service scanning,Metasploitable,Reconnaissance,OS Fingerprinting,Windows 7,TheftDoS/DDoS TCP, DoS/DDoS UDP,Ubuntu Tap,DoS/DDoS HTTP,ExternalKeylogging, data exfiltrationThrough rigorously analyzing the evaluation datasets, we figured out that the number of samples of each attack type is slightly imbalanced; thus, employing resampling methods is necessary. In our experiments, we experimented and evaluated the Random Undersampling algorithm (RU) and its conjunction with Synthetic Minority Oversampling Technique (RU-SMOTE).(i)
Random Undersampling is a random selection process running on overwhelmed attack types to reduce their size. However, randomly selecting data points might accidently ignore critical information, resulting in degraded classification performance.(ii)
Synthetic Minority Oversampling Technique (SMOTE) balances the dataset by synthesizing new samples for the minority class. In more detail, it selects a cluster of samples and draws a line between them; new samples are the points along this line.To implement these resampling methods, we utilizedimblearn library [33] supporting multiple resampling techniques along with several running strategies for both binary and multiclass classification. In binary classification, we need to configure the ratio between minority class and majority class after resampling, whereas this configuration is unnecessary in multiclass classification. From the results illustrated in Figures 4-6, it is obvious that these resampling techniques have no significant impact on the performance of models. Furthermore, if the number of samples in majority classes drastically outweighs ones of minority classes, they may decrease the attack detection quality of the smaller classes. Therefore, our models are trained without resampling techniques.Figure 4
The macroaverageF1-score of the IoTID20 dataset.Figure 5
The macroaverageF1-score of the CIC-IDS-2017 dataset.Figure 6
The macroaverageF1-score of the BOT-IoT dataset.


## 4.3. Results and Discussion

To select classification algorithms for our multistage IDS, we examined the detection time and accuracy of several supervised machine learning algorithms on the IoTID20 dataset. In detail, the ability to classify benign and malicious network traffic (binary attack detection) and identify precisely the types of attacks (multiclass attack detection) are both considered. Since operating on network gateways requires a lightweight attack detection model, experimented algorithms are simple machine learning algorithms, including linear support vector machine, quadratic support vector machine, K-nearest-neighbor, linear discriminant analysis, quadratic discriminant analysis, multilayer perceptron, long short-term memory, autoencoder classifier, and decision tree classifier; their results are presented in Table6. As shown in the table, the decision tree classifier outperforms other algorithms, and it is considered a lightweight machine learning algorithm [34]. This classifier is thus selected for the attack detection model of MidSiot. Note that empty values in the table (denoted by N/A) imply the long training time exceeding two hours. Moreover, training a decision tree classifier is trivial and possibly performed on IoT devices. We also applied Classification and Regression Tree (CART) to boost the detection performance. In more detail, CART splits training data into two subsets based on a specific feature k and a threshold tk (e.g., “flow duration ≤ 100”). This split is repeated on each subset until it reaches the maximum depth or subset size equals to 0. As a result, the computational complexity of the classifier is reduced to Olog2m with m being the number of samples in the training set. This significantly increases the training and prediction rate to deal with large datasets.Table 6
The results of examined classifiers.
ModelClassificationAccuracy (%)Training timeμs/flowPrediction timeμs/flowAdjusted parametersLinear support vector machineBinary98.161150.57204.54kernel = lineargamma = autoMulticlassN/A10 298.31453.1Quadratic support vector machineBinary98.25792.61311.5kernel = polygamma = autoMulticlassN/AN/AN/AK-Nearest neighborBinary99.790.172343.72n_neighbors = 5Multiclass98.610.192377.81Linear discriminant analysisBinary95.0721.60.27All defaultMulticlass80.7325.714.35Quadratic discriminant analysisBinary53.618.8912.44All defaultMulticlass56.6216.6214.74Multilayer perceptronBinary99.62.570.65Input layer and first layer with 50 neurons and activation = reluOutput layer with activation = sigmoidMulticlass92.714.947.96Long short-term memoryBinary96.53572.4462.64input layer and LSTM layer with 50 neuronsOutput layer with activation = sigmoidMulticlassN/AN/AN/AAutoencoder classifierBinary93.0111.650.62Encoding layer with 50 neurons and activation = reluDecoding and output layer with activation = softmaxMulticlass87.7413.350.81Decision tree classifierBinary99.9412.490.43criterion = entropyMulticlass99.6916.760.38Attack detection results: Table 7 reports the overall performance of each MidSiot’s stage on evaluated datasets. We can see that MidSiot could not only accurately differentiate between normal and malicious traffic in the second stage, but also identify the type of attacks in the third stage. The average accuracy of such stages is about 99.98% and 99.68%, respectively. Regarding classifying IoT devices, our proposal achieved a high classification accuracy reported at 95.55% on average. In detail, device-type classification for BOT-IoT achieves the highest result, at 99.92%, whereas the results of IoTID20 and CIC-IDS-2017 are about 92.57% and 94.17%, respectively. To have a better understanding the attack detection performance of MidSiot, Tables 8–10 illustrate the confusion matrices, which present the comparison between predicted attacks and the actual ones.Table 7
The evaluation results of the multistage IDS.
DatasetStageAccuracy%Training timeμs/flowPrediction timeμs/flowIoTID20192.5735.760.44299.9511.330.26399.1515.90.31CIC-IDS-2017194.17110.310.89299.9925.650.25399.9778.680.27BOT-IoT199.9219.270.15299.993.240.07399.9315.050.17Table 8
Attack detection confusion matrix on the IoTID20 dataset.
PredictedMiraiScanDoSNormalMITM ARP spoofingActualMirai4124462583038569Scan61874 47309155DoS925935653Normal631565330MITM ARP Spoofing62420021634509Table 9
Attack detection confusion matrix on the CIC-IDS-2017 dataset.
PredictedBenignDoS/DDoSPortscanBotWeb attackActualBenign15830362735DoS/DDoS19235881802Portscan6615891101Bot100019290Web Attack401201463Table 10
Attack detection confusion matrix on the BOT-IoT dataset.
PredictedDDoSDoSReconnaissanceNormalActualDDoS192618943500DoS131983078600Reconnaissance26818182Normal1132Comparing with baseline methods: We compared attack detection quality between our proposal and state-of-the-art IDS and reported the results in Table 11. Overall, MidSiot outperforms its competitors on CICIDS-2017 and BOT-IoT datasets and is comparable with them on IoTID2020 dataset. In more detail, regarding CIC-IDS-2017, the best of our competitors achieves 99.9% in both binary and multiclass classification, whereas our proposed IDS achieves better results recorded about 99.99% and 99.97%, respectively. Similar results are also found in the BOT-IoT dataset, in which our proposal achieves 99.99% accuracy in binary classification and 99.93% in multiclass classification problems. In the IoTID20 dataset, the best competitor detects the attack types with 100% accuracy, and MidSiot also has very competitive results reported at 99.15%. Compared with state-of-the-art IDSs, MidSiot employed more machine learning models to enhance detection accuracy. This demands high computation costs and training datasets to train these models. Indeed, each stage in MidSiot has a different training dataset, which requires a huge effort to label. For example, the first-stage model needs to label the device type of network traffic, whereas the second-stage model demands labeling abnormal traffic. Moreover, deploying the first-stage model from the cloud to IoT local gateway consumes network bandwidth and may trigger delays.Table 11
Comparing the proposed system with other one-stage systems.
DatasetWorkAttack detection modelBinary attack detection%Multiclass attack detection%Prediction timeμs/netflowCICIDS-2017MidSiotMultistage99.9999.971.41Gamage and Samarabandu [35]Random forestN/A99.8660.48Vinayakumar et al. [36]Deep neural network93.1095.60N/AElmrabit et al. [37]Decision tree99.9099.90N/AManimuruganDeep belief network99.3797.73N/AIOTID20MidSiotMultistage99.9899.881.01Ullah and Mahmoud [21]Decision tree99.9499.69N/AAlkahtani and Aldhyani [38]Long short-term memory98.20N/AN/ASong et al. [39]Autoencoder93.7695.20N/AHussein et al. [40]Random forest99.9099.90N/AUllah and Mahmoud [41]Convolution neural network99.9897.76N/AIslam et al. [42]Decision treeN/A100.001139.37BOT-IOTMidSiotMultistage99.9999.990.39Ferrag et al. [43]Rules and decision treeN/A97.001.54Ferrag et al. [44]Deep autoencoderN/A98.391916.55Dwibedi et al. [45]Support vector machine99.99N/AN/APokhrel et al. [46]K-Nearest neighbor92.10N/AN/AGe et al. [47]Support vector machine99.7499.03693 040Ullah and Mahmoud [41]Convolution neural network99.9099.97N/AIn conclusion, by using a hierarchical architecture and chaining stages together, MidSiot effectively classifies device types, identifies abnormal network traffic, and differentiates cyberattack types.


## 5. Conclusion

In this article, we proposed a distributed intrusion detection system for IoT scenarios, in which connected devices are not only resource-constraint but also heterogeneous in hardware specification. To accurately detect various types of cyberattacks, the proposed IDS consists of three stages: (1) classifying device types; (2) detecting malicious network flows; and (3) identifying attack types. In the experiments on three popular IOT-IDS datasets (IoTID20, CIC-IDS-2017, and BOT-IoT), we demonstrated that our proposal could detect several attacks with an accuracy of 99.68% on average and outperforms state-of-the-art IDSs. In addition, we examined two resampling techniques to balance the datasets and discovered that these techniques slightly reduce the detection rate of minority attack types. In short, MidSiot is beneficial for both the industrial and research communities interested in further developing intrusion detection systems for IoT.



---

*Source: 9173291-2022-02-21.xml*