Inline high‐bandwidth network analysis using a robust stream clustering algorithm

IET Information SecurityVolume 13, Issue 5 p. 486-495 Research ArticleFree Access Inline high-bandwidth network analysis using a robust stream clustering algorithm Morteza Noferesti, Morteza Noferesti Data and Network Security Lab, Sharif University of Technology, Tehran, IranSearch for more papers by this authorRasool Jalili, Corresponding Author Rasool Jalili jalili@sharif.edu Data and Network Security Lab, Sharif University of Technology, Tehran, IranSearch for more papers by this author Morteza Noferesti, Morteza Noferesti Data and Network Security Lab, Sharif University of Technology, Tehran, IranSearch for more papers by this authorRasool Jalili, Corresponding Author Rasool Jalili jalili@sharif.edu Data and Network Security Lab, Sharif University of Technology, Tehran, IranSearch for more papers by this author First published: 01 September 2019 https://doi.org/10.1049/iet-ifs.2018.5287Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract High-bandwidth network analysis is challenging, resource consuming, and inaccurate due to the high volume, velocity, and variety characteristics of the network traffic. The infinite stream of incoming traffic forms a dynamic environment with unexpected changes, which requires analysing approaches to satisfy the high-bandwidth network processing challenges such as incremental learning, inline processing, and outlier handling. This study proposes an inline high-bandwidth network stream clustering algorithm designed to incrementally mine large amounts of continuously transmitting network traffic when some outliers can be dropped before determining the network traffic behaviour. Maintaining extended-meta-events as abstracting data structures over a sliding window, enriches the algorithm to address the high-bandwidth network processing challenges. Evaluating the algorithm indicates its robustness, efficiency, and accuracy in analysing high-bandwidth networks. 1 Introduction Numerous network services, from real-time user monitoring to providing network security, require real-time high-bandwidth network traffic analysis. Growing in size and complexity of networks makes administrators using more practical and intelligent network traffic analysis tools to maintain the network more stable, available, secure, and well-planned. Continuous increasing of applications in high-bandwidth networks, which leads to high-volume, velocity, and variety of traffics, initiates new technical challenges and requirements in traffic analysis systems. The port-based analysis is the first classical method, which analyses the network traffic based on the well-known port numbers. Moore and Papagiannaki [1] claim that port-based methods have no better accuracy than 70%. Madhukar and Williamson [2] show that port-based methods are unable to identify 30–70% of internet traffic flows they investigated. In a recent study [3], Alcock et al. quantify network traffic on major transmission control protocol (TCP) and user datagram protocol (UDP) ports and confirm that port-based identification is ineffective. An alternative to port-based analysis approaches is traffic packets inspection methods, known as deep packet inspection (DPI) techniques. Such methods infer application types by looking for application-specific data within traffics' payload. These approaches rely on two related assumptions, visibility of payloads and being known the syntax of application packets payload. Although DPI-based analysis is claimed to achieve high accuracy, the assumptions are not valid in real situations. The visibility of payloads assumption can be violated when the traffic is encrypted. The second assumption produces a heavy operational load, especially in high-bandwidth networks. Moreover, updating and changing in each application reduce accuracy. Given the high resource requirements, currently, DPI-based methods are used to analyse network traffic in small networks and to establish accurate ground-truth data. The low accuracy of port-based analysis methods and the high overhead of DPI-based analysis methods have motivated the use of machine-learning techniques for traffic analysis [4]. Current machine-learning-based methods can be structured into supervised and unsupervised. The supervised methods detect characteristic patterns of different applications, utilising a labelled dataset [5]. The unsupervised learning approaches cluster traffic flows into groups that have similar traffic patterns. Clustering traffic is widely utilised in application-specific services, such as efficient network resource management [6] or applying different tariffs for provided services [5], as well as the network security domain, such as automated intrusion detection systems [7] or detection of denial of service attacks [8]. Basically, three main clustering methods have been used in the network traffic clustering literature [5, 9-11]: (i) the classic k -means algorithm that partitions instances into disjoint clusters, (ii) the clustering algorithms that generate a hierarchical grouping of instances, (iii) the probability based method assigns instances into classes probabilistically, not deterministically. High-bandwidth networks are dynamic environments having real-time incoming data forming a potentially infinite stream with unexpected changes. Such environments require adaptive analysing approaches which learn the high-bandwidth network traffic dynamic behaviours. The analysing approaches should satisfy the following [12-14]: Incremental learning: The methods should be periodically updated to adapt to new applications. Outlier handling: High-bandwidth networks tend to be noisy due to the lack of prior processing. Analysing methods should have strategies to detect and resolve outliers. Forgetting mechanism: Analysing methods have to forget the expired learning information. Inline processing: Unbounded high-bandwidth network traffic imposes that each packet to be processed once (sometimes called one-pass processing restriction). In this study, the high-bandwidth network traffic is modelled as a massive and unbounded sequence of flows, continuously generated at rapid rates, called a flow stream (FS) [15-17]. Successive packets having the same five-tuple 〈source-IP, source-port, destination-IP, destination-port, TCP/UDP protocol〉 construct a flow. A FS is defined as Definition 1.A FS is a potentially infinite ordered sequence of flows, denoted by where all flows follow the schema and are flow attributes in a d -dimensional space. The FS clustering problem is defined as maintaining a consistent set of clusters contain the sequence of observed flows so far, in a bounded amount of memory and time. More formally: Definition 2.Suppose that is a sequence of observed flows of an input FS at time-stamp t. A stream clustering algorithm should find a set of clusters , where and . Considering the high-bandwidth network traffic as a dynamic environment having a potentially infinite stream of flows, this study proposes a high-bandwidth network stream clustering algorithm (HBN-SCA) as a window-based stream clustering algorithm to tackle high-bandwidth networks analysing challenges such as incremental learning, outlier handling, forgetting mechanism, and inline processing. Under the sliding window model, the algorithm maintains statistics and information about the recent traffic in an abstract data structure called extended-meta-event (EM). The EMs are used to increase the clustering accuracy and robustness against traffic outliers. Experimental results have empirically demonstrated that developing HBN-SCA yields high accuracy and efficiency in real network traffic and workloads. 1.1 Contributions The main contribution of the study is to cluster the high-bandwidth network traffics in a robust setting where outliers are detected in a systematic approach. To handle outliers properly, HBN-SCA models the traffic of the high-bandwidth network as a noisy stream in a formal way and uses the algorithm proposed by Zarrabi-Zadeh et al. [18] to analyse the input stream where the specific number of outliers can be dropped from the stream before analysing. Other contributions of this study are as follows: The main difficulty in handling outliers in the data stream model is the absence of any information about the incoming flows in the future. We define the EM structure as a summarisation of a set of d -dimensional flows based on the meta-event data structure, defined in [14]. The EMs have a notion of a buffer, intended to handle outliers in the algorithm. The outliers would be caught in the buffer or left outside of the already existing EMs and put in a global buffer called window. Creating new EMs by finding its centre and radius is very important in the accuracy and performance of the algorithm. HBN-SCA uses the density-based spatial clustering of applications with noise (DBScan) algorithm [19] to create new EMs based on the density of flows in the window. Accordingly, HBN-SCA detects core-flows in the window and dynamically finding the EM radius. 1.2 Organisation The rest of this paper is organised as follows: an overview of practical methodologies employed in high-bandwidth network analysing is given in Section 2. HBN-SCA is defined in Section 3. Empirical simulation and real-network implementation of HBN-SCA are explained in Section 4 where it is demonstrated that HBN-SCA produces accurate clusters regarding the input FS. Finally, the on-going challenges and future directions are discussed in Section 5. 2 Related works Related works are surveyed in two subsections. Firstly, the state-of-the-art in high-bandwidth network analysing is discussed. Then, the current stream clustering algorithms are reviewed. 2.1 High-bandwidth network analysing approaches The specific application of machine-learning techniques to traffic classification has attracted large attention from the research community [9, 11]. Perera et al. [5] surveyed the literature of machine-learning techniques in the field of network traffic classification. They compared the performance of six widely-used supervised machine-learning algorithms such as multilayer perceptron, decision tree, random forest, Bayes networks, naive Bayes, and Bayes tree for classifying network traffic. The evaluations were conducted for the classification of five distinct network traffic classes and two feature selection techniques. Their results indicated that the random forest and decision tree algorithms are promising classifiers for network traffic in terms of both classification accuracy and computational efficiency. In recent years, continuous increasing of network bandwidth dictates new requirements to be addressed in network analysing methods. The U.S. patent [20] focusing on real-time network monitoring and security concludes that security systems such as firewalls, anti-viruses, and intrusion detection systems require real-time high-bandwidth network processing. While the patent provides the commercial impacts of research about real-time high-bandwidth network monitoring and security; it advises that the full-line rate of a high-bandwidth network corresponds to rates above 100 Mbps and in some cases above 1000 Mbps. Noferesti and Jalili [14] proposed a behaviour detection system, namely high-bandwidth network behaviour detection system (HB2 DS), where a data structure called meta-event is defined whose inter-relationships are used to mine end-user behaviours. Authors represented a formal definition of a behaviour detection system and demonstrated that HB2 DS satisfies the behaviour detection system constraints. One group of researchers attempts to distinguish unknown applications in high-bandwidth networks. Zhang et al. [4] proposed the robust traffic classification method by combining supervised and unsupervised learning techniques to identify unknown applications. In [21], Wang et al. proposed the unclean traffic classification (UTC) method, which filters unclean training samples. UTC applies multiple classifiers in the training phase. If most classifiers mark a data as noise, it will be removed from the training data. Lin et al. [22] proposed an unknown network protocol classification method based on semi-supervised learning. They apply the K -means algorithm over training data. For each cluster, if it contains no labelled sample, it is detected as unknown. The other group of researchers attempts to increase the performance of machine-learning methods in network analysis. Arora et al. [23] used Netflow to categorise network nodes in a data centre. Based on Netflows' information, nodes are divided into either the client or the server. Lu et al. [24] used message size sequence and distribution to classify some popular network protocols, such as file transfer protocol (FTP) or hypertext transfer protocol (HTTP), in high-bandwidth networks. Gomes et al. [25] surveyed the network traffic analysis approaches using ensemble learning methods. Utilising these methods, Casas et al. [12] combined multiple DBScan algorithms to increase the network traffic clustering accuracy. They divided the feature space into sub-spaces and applied the DBScan algorithm on each subspace. Then, a voting algorithm distinguished anomalies in network traffic. Jin et al. [13] presented a series of simple linear binary classifiers, each of which can be efficiently implemented and trained in parallel. 2.2 Stream clustering algorithms Data stream mining is an active research area that has recently emerged to discover knowledge from large amounts of continuously generated data [26, 27]. Major data stream clustering approaches can be categorised into partitioning algorithms, micro-clustering algorithm, grid-based algorithms, model-based algorithms, and density-based algorithms. Density-based clustering can discover the clusters of non-spherical shape, filter outliers, and have no assumption on the number of clusters [28]. So, density-based clustering is more appropriate for high-bandwidth network clustering and is reviewed in this section. D-Stream [29] is an algorithm for clustering data streams based on the density-based approach. It has online and offline phases. The online phase maps the data into some grids and the offline phase clusters the grids based on their densities. It uses fading window model to capture the evolution of clusters. It assigns a weight to each data point and determines grid density based on these weights. D-Stream forms a cluster based on high-density grids which are called dense grids. Final clusters are formed by merging dense grids. Furthermore, it has a technique for removing the sporadic grids mapped by outliers. DenStream [30] is also a density-based method for clustering an evolving data stream. It has online and offline phases; the online phase uses core-micro-clusters, potential-micro-clusters, and outlier micro-clusters to summarise clusters with arbitrary shapes and distinguish potential outliers. The offline phase clusters the potential micro-cluster based on DBScan. Moreover, a pruning strategy is designed to remove the outlier micro-cluster periodically. CluStream [26] is designed to cluster data streams over different time horizons in an evolving environment. Since the snapshots of clustering results over the landmark window have to be stored, CluStream consumes more space, and so is not suitable for the clustering problem over sliding windows. In addition, the influence of expired records cannot be promptly eliminated during on-line clustering, while such a prompt elimination is required in sliding window clustering. Aggarwal [15] extended cluster feature for sliding window model to present cluster distribution changes as well. The proposed approach keeps aggregates over the sliding window and calculates clustering based on all exponential histogram cluster feature synopsis. The approach cannot discover clusters with arbitrary shapes and does not have the ability to handle outliers. SDStream [31] extends it for density-based clustering. The algorithm uses the potential and outlier micro-cluster concepts and keeps them in the form of an exponential histogram. SDStream removes the outlier micro-clusters, which are not in the sliding window. While state-of-the-art in network traffic analysis try to improve accuracy and/or performance, the dynamic nature of high-bandwidth networks is not considered. Clustering of dynamic network traffic requires that a process be able to incrementally cluster flows within memory and time restrictions [16]. In addition to evaluating the performance of stream clustering algorithm in analysing network traffic, this study proposes HBN-SCA as a high-bandwidth network specific clustering algorithm which has incremental learning, outlier handling, forgetting mechanism, and inline processing properties. 3 High-bandwidth network stream clustering algorithm (HBN-SCA) To satisfy the HBN analysing requirements, we introduce HBN-SCA. Considering as a high-bandwidth network FS, O as the set of outliers, and as the set of non-outliers. In [14], the authors use the well-known minimum enclosing ball problem to maintain network traffic behaviours. The problem is to fit a ball over FS, where each flow processed once. The approximation algorithm finds the region R (FS) enclosing flows in FS which is factor of the optimum area . Considering outliers in the FS, an -approximation algorithm for the problem satisfies the followings: (1)The upper bound for is [14]. HBN-SCA uses the algorithm proposed in [18] to provide some upper bounds on for high-bandwidth networks with outliers. The main difficulty in handling outliers in the data stream model is the absence of any information about the incoming flows in the future. HBN-SCA uses a buffering framework, which postpones the decision about the input flows until it gets enough information. The buffering framework is based on the EM data structure which is defined as follows: Definition 3.An EM is a summarisation of a set of d -dimensional flows ; which is defined as a tuple , where (the meta-event centre) corresponds to a vector of d -entries, r (radius) is the maximum distance of flows from , b is a buffer containing at most m flows of the stream, n is the number of flows, and t is a time-stamp declaring the last update of the EM. For each dimension, the average of attributes is maintained in . The entry of is equal to , where is the value of in the dimension. To cope with the non-deterministic nature of the input stream, the EM data structure has two important properties: Incrementality: An incoming flow can be easily inserted into an EM by updating its attributes. Considering as a distance function and , the new radius and centre of EM can be calculated according to (2) and (3) (2) (3) Additivity: Two disjoint EMs, and , can be easily merged into . In [14], it is shown that is obtained by inserting a flow into regarding the incrementality property, where HBN-SCA consists of two main steps: online abstraction and offline conclusion. The online abstraction step maintains a set of EMs at time-stamp t. It has two main modules, namely Learning and Window. The Learning module processes the input FS inline and incrementally updates the stream status in . The Window module defines new EMs over the current state of the input stream. The Conclusion module in the offline abstraction step constructs clusters set using . 3.1 Online abstraction HBN-SCA summarises incoming flows in an online abstraction step. Fig. 1 shows an example of the abstraction step procedure. The total stream is depicted in part (1). In part (2), flows which are indicated by black colour, are inserted into the algorithm. As the figure shows, HBN-SCA tries to cover the new arriving flows with EMs (circles). As the stream proceeds, part (3), some EMs are merged (the additivity property), and some are updated regarding the arriving flows (the incrementality property). Regarding these two properties, HBN-SCA can determine the arbitrary-shaped clusters. In the next two subsections, first, we show how the Learning module processes the input FS and incrementally update the abstraction. Then, the details of the Window module will be discussed. Fig. 1Open in figure viewerPowerPoint Example of the online abstraction step of HBN-SCA 3.1.1 Learning module HBN-SCA maintains an EM set, , which contains all the EMs detected by the algorithm until time-stamp t. For each input flow, HBN-SCA decides to insert it into an EM buffer or to put it into the window. If the input flow inserted into the buffer and it is made full, the algorithm extracts one flow from the buffer and updates the EM. Now, the problem is how to extract flows from the buffer in such a way that only outliers remain in the buffer. In an ideal scenario, all outliers are kept in the buffer, and only non-outlier flows are extracted. However, this is impractical in real networks as outliers and non-outliers are not precisely distinguishable. HBN-SCA provides a mechanism to bind the total error based on the algorithm proposed in [18]. The algorithm uses the concept of centre-point, which is defined as follows: Definition 4.Considering an n -point set P in d dimensions, a point is called a centre-point of P if any half space containing c contains at least points of P. In other words, any half space that avoids a centre-point can contain at most flows of FS. Considering FS with size where z is the number of outliers, and c is the centre-point of FS, regarding Definition 4, any region that avoids c contains at most the following number of flows: (4)From the other side, the number of non-outliers in FS is at least: (5)Regarding (4) and (5), c is contained in any region than encloses non-outliers, and in particular, is contained in the final minimum enclosing region. So, let b be the buffer of an EM, if , HBN-SCA extracts from b a flow which is closest to the centre-point of b. Zarrabi-Zadeh and Mukhopadhyay [18] proved that this algorithm yields an approximation factor of . To achieve the approximation factor, the buffer size (m in Definition 3) should be more than . The details of the HBN-SCA learning module is described in Algorithm 1 (see, Fig. 2). HBN-SCA reads flows continuously from the FS. All the EMs detected by the algorithm is stored in a set called . W is the window data structure. The value of and represent the outlier detection and expired threshold. min-Point, eps, and StepSize are used in the window processing algorithm. Fig. 2Open in figure viewerPowerPoint Algorithm 1: HBN-SCA (FS, , W, , , min-Point, eps, StepSize) According to Algorithm 1 (Fig. 2), if there exists an EM in EMSt which covers the flow (the flow distance to the EM centre is less than the EM radius), it is inserted into the EM buffer. When the buffer becomes full, the algorithm finds the buffer's centre point according to Definition 4. Then, the closest flows to the centre-point in the buffer are inserted into the EM. If there is not any EMs covering the input flow, the flow is inserted into the window (W). When the window becomes full, ProcessWindow algorithm is invoked [Algorithm 2 (see Fig. 3)] to generate new EMs, which are inserted into . Finally, the window W set free. Fig. 3Open in figure viewerPowerPoint Algorithm 2: ProcessWindow (W, , eps, min-Point, TempExtendedMetaEvent, StepSize) The HBN-SCA forgetting mechanism specifies how old data are discarded. It is a trade-off between the reactivity of the system and robustness to outliers. HBN-SCA uses a gradual forgetting mechanism based on a linear decay technique. EMs have a time-stamp shows its last update (e.g. insert a new flow or merge with another EMs). If one EM does not update for more than the threshold, it is expired and removed from . 3.1.2 Window module To summarise the continuously-arriving flows and, at the same time, giving the greater importance to up-to-date ones, a time window is defined in HBN-SCA, which covers the most recent flows, which are not covered by EMs. More formally, the window W consists of a set of flows , where all flows are not assigned to an EM in . The window model is similar to the sliding window model, which is widely adopted in stream mining [25]. HBN-SCA creates new EMs based on the existing flows in W. It also detects outliers according to the time quantum that flow remains in the window. To process the window, the concept of core-flow is defined as follows: Definition 5.(core-flow): For an eps and a min-Point value, a flow in a window W is a core-flow, if there exists at least a minimum number (min-Point) of flows such as in W where . Fig. 4 presents the different flow distributions in a window when the stream proceeds. In the figure, the window size and min-Point values are 10 and 3, respectively. In part (1) of the figure, the algorithm finds a core-flow while there exists min-Point number of flows where their distance to the core-flow is less than eps value. Another distribution is shown in part (2), where the algorithm is unable to find a core-flow, so it increases the eps value until finding a core-flow. The increased ratio is determined by StepSize initial parameter. Fig. 4Open in figure viewerPowerPoint Different flows distributions in a window The details of the window processing algorithm, ProcessWindow, is represented in Algorithm 2 (Fig. 3). The algorithm adapts eps and uses the DBScan algorithm to find core-flows. Then, it defines EMs for core-flows. To define an EM (), its parameters are initialised, where is the core-flow, the initial value is set to eps, is set to the size of W divided by the size of EMSt, and are set to zero and the current time-stamp, respectively. These new EMs are inserted into a set, namely TempExtendedMetaEvent. For each flow in the window, if there exists an EM in TempExtendedMetaEvent, which covers it, the flow is removed from the window. If a flow remains in the window (there exist no similar flow to it) more than threshold, it is marked as an outlier. HBN-SCA detects outliers based on the time period they remain in the window. The threshold specifies the time threshold one flow can remain in the window. When the threshold is overcome, the flow is considered as an outlier. In Fig. 4 part (2), supposing , the flow is labelled outlier because it remains in a sparse region for two windows. 3.2 Offline conclusion Utilising EMs as the summarised data structures generated in the online abstraction step, HBN-SCA tries to detect the related EMs and insert them into the same clusters. Suppose and as two EMs, if their distance is less than the sum of their radius, HBN-SCA inserts them into the same cluster. The offline conclusion step is periodically invoked, which produces convex or non-convex clusters over the current state of the input stream. 3.3 Complexity analyses The complexity of the online and offline phases, as well as the complexity of window processing of HBN-SCA, are shown in the following theorems, where m is the number of EMs in , n is the number of clusters in the cluster set , and w is the window size. Theorem 1.The time complexity of the process window algorithm is . Proof.The algorithm uses the DBScan algorithm over the window with size w, which has the time complexity . HBN-SCA executes DBScan to adapt eps value. In the worst case, DBScan executes diameter (W)/StepSize times. diameter(W) is the diameter of the window. While StepSize is a constant value, the process window algorithm time complexity is . □ Theorem 2.: The time complexity of the online abstraction phase is . Proof.The online abstraction phase contains two modules. The first finds the closest EM for each input flow, so its time complexity is , where m is the number of EMs. It also updates the EM where it requires computing of centre-points of flows in d dimensions which is polynomial in d and z. The second module processes window algorithm where its complexity is . As a result, the time complexity of online abstraction phase is . □ Theorem 3.The time complexity of the offline conclusion phase is . Proof.The offline conclusion tries to detect the related EMs and inserts them into the same cluster of . So, the algorithm should compare all the entities in the EM set pairwise. That sets the time complexity of the offline conclusion phase to . □ 4 Evaluation To evaluate our proposed framework in terms of analysing high-bandwidth networks, we developed HBN-SCA and compared its results with HB2 DS [14], DBStream [32], and DenStream [33]. The approaches are implemented in C++ (gcc version 5.2.1) on Intel Xeon E5620 CPU having Linux kernel version 3.17.0.3, and 42 GB of main memory. To read packets directly from the interface drivers, we used the PF_RING library [34] (version 6.4.1), which provides direct access to traffic packets without the kernel inter-mediation. 4.1 Evaluation criteria To evaluate the quality of detected clusters, four evaluation criteria are used: purity, Rand index (RI), sum of squared errors (SSE), and Silhouette. Purity and RI parameters are used to measure the similarity of flows inside a cluster. SSE parameter and Silhouette coefficient represent the similarity of a flow to i