Sketch‐BF: A fast algorithm for finding top‐k flows

Electronics LettersVolume 58, Issue 11 p. 429-431 LetterOpen Access Sketch-BF: A fast algorithm for finding top-k flows Jie Lu, Corresponding Author Jie Lu [email protected] orcid.org/0000-0002-6831-7413 PLA Information Engineering University, Zhengzhou, China Email: [email protected]Search for more papers by this authorHongchang Chen, Hongchang Chen National Digital Switching System Engineering and Technological Research Center, Zhengzhou, ChinaSearch for more papers by this authorZheng Zhang, Zheng Zhang PLA Information Engineering University, Zhengzhou, ChinaSearch for more papers by this authorZhen Zhang, Zhen Zhang PLA Information Engineering University, Zhengzhou, China Network Communication and Security Purple Mountain Laboratory, ChinaSearch for more papers by this authorLiang Wang, Liang Wang PLA Information Engineering University, Zhengzhou, ChinaSearch for more papers by this author Jie Lu, Corresponding Author Jie Lu [email protected] orcid.org/0000-0002-6831-7413 PLA Information Engineering University, Zhengzhou, China Email: [email protected]Search for more papers by this authorHongchang Chen, Hongchang Chen National Digital Switching System Engineering and Technological Research Center, Zhengzhou, ChinaSearch for more papers by this authorZheng Zhang, Zheng Zhang PLA Information Engineering University, Zhengzhou, ChinaSearch for more papers by this authorZhen Zhang, Zhen Zhang PLA Information Engineering University, Zhengzhou, China Network Communication and Security Purple Mountain Laboratory, ChinaSearch for more papers by this authorLiang Wang, Liang Wang PLA Information Engineering University, Zhengzhou, ChinaSearch for more papers by this author First published: 08 April 2022 https://doi.org/10.1049/ell2.12482AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Finding top-k flows plays an essential role for many network managements. Since the line rates keep increasing in networks, finding top-k flows precisely and quickly becomes more challenging. Existing algorithms for top-k finding can achieve high precision suffering from slow speed. In this paper, the authors propose Sketch-BF, which is a novel top-k finding solution that is much faster than existing algorithms while ensuring high precision. The core idea is to add a bloom filter variant between sketch and heap to accelerate by significantly reducing the min-heap update. The authors' experimental results show that, while maintaining precision, Sketch-BF has about two to five times higher insertion speed compared with the state-of-the-art. Introduction Finding top-k flows in the data stream is fundamental research in network measurement. Many network managements, such as traffic engineering, traffic scheduling, anomaly detection, benefit from top-k finding. However, it is a challenge to track top-k flows accurately and fast in modern high-speed networks. Because of memory limitation, the usable memory is too small to record all flow, especially when there are many other network functions in the monitor. Besides, the line rate in a high-speed network consistently exceeds 10 Gb/s, making it almost impossible to track the information of all flows accurately. The research on top-k flow can be divided into the record-some method and the record-all method based on the tracking object. The record-some method only tracks partial flows to reduce memory usage. When encountering a different flow whose frequency just increases and gets itself in cells, this method evicts the minimum flow from cells. And the counter of this new flow would be set as minimum value +1. The critical insight of this method is that top-k flows would occur much more than other flows. The typical examples are space-saving algorithm [1] and Lossy counting [2]. Based on that, the researcher presents some improvement approaches. For instance, HeavyKeeper [3] and WavingSketch [4] adopt an exponential decay update strategy to improve count accuracy, and ActiveKeeper [5] uses an exponential counter to reduce memory. The record-all method tracks all flow frequency using compact probability data structure and heap to record top-k flows, such as count-min heap (CMH) [6], conservative-update heap [7], count heap [8]. The basic framework of this method is composed by sketch-part and heap-part. The sketch-part in the method estimates the frequency of all flows, and the heap-part stores the key and value of top-k flows. Many sketch variants [9-11] have been developed to improve the precision of flow frequency. However, to keep min-heap correctly, the record-all method must update min-heap simultaneously when getting an estimated frequency of new incoming packets. Thus, its process will be compromised because of the slow heapify operation. To solve the problem left by prior work, we propose an efficient top-k finding solution called Sketch-BF which outweighs the past work in speed. The key insight is most min-heap updates increase the count without flow replacement in the record-all method. If we update min-heap only when the estimated frequency of a new flow exceeds the minimum frequency in min-heap, the processing speed would be significantly improved. Our main contributions are shown as follows: We add bloom filter to the record-all method, and propose Sketch-BF which achieves high precision and fast speed for finding top-k flows. We conduct extensive experiments on real network streams and results show that Sketch-BF has about two to five times higher insertion speed than the state-of-the-art while maintaining a high precision in top-k finding. The design of Sketch-BF Problem statement The network traffic in an observation window is viewed as a set of M flows = {f1,f2,…, fM}, where each flow fi is composed of packets sharing the same flow key, e.g. source internet protocol (IP). Flow frequency refers to the number of packets belonging to this flow. Let ni and n ̂ i ${\hat{n}_i}$ represent the real frequency and estimated frequency of flow fi, respectively. Given an integer k, top-k finding methods report a list of k flows from F with the estimated largest flow frequencies. The naive record-all method The naive record-all method is composed of two stages. The sketch in the first stage uses many counters to store the flow frequency. For each incoming packet, the sketch would return its estimated frequency and key as the input of the second stage. According to the estimated frequency and key, the second stage uses a heap to track top-k flows. The main problem with using heap to store top-k candidate flows is hard to ensure whether a flow has been stored in a min-heap. This problem exists in both record-all and record-some methods. For example, assume flow fi has been stored in the min-heap with value 17 and the min value in min-heap is 10. A new packet belonging to flow fi into the sketch gets the estimated value 18. Because the estimated value is larger than 10, we need to traverse min-heap. If flow fi has been stored in the heap, we need to update the value, or we need to replace the minimum flow with flow fi, and heapify the heap. In the data stream, the packets are processed one by one in order, and the estimated flow frequency is constantly updating and increasing. And the heap itself does not provide a check mechanism for the uniqueness of heap elements. Therefore, a traverse of the heap to keep uniqueness is necessary but slow. The design of Sketch-BF To reduce the consumption of finding the uniqueness of heap elements, we introduce a bloom filter variant in the record-all method to quickly judge whether a flow is stored in the heap. Besides, we reuse the sketch's hash to reduce the computational overhead of the bloom filter's hash. Our sketch-BF, as shown in Figure 1, adds a bloom filter variant between sketch and heap. We use an example to demonstrate the sketch-BF design and its benefits. Suppose we have a new incoming packet P89 belonging to flow f4. Then we increment the associated counter with 1 and return 32 as the estimated frequency of flow f4. Next, the bloom filter judges that flow f4 is a member in a heap because bits in all corresponding positions are 1. Therefore, we do not need to update the heap. Fig. 1Open in figure viewerPowerPoint The data structure of Sketch-BF As shown in Figure 1, sketch and bloom filter in Sketch-BF both comprise d arrays, and each array consists of w counters. The difference between sketch and bloom filter is that the counter in sketch records the number of packets while that in bloom filter maintains 1 bit to check whether a flow is a member of heap. The counter of the bloom filter corresponds to the counter of the sketch one-to-one so that we can share the hash result. The min-heap stored the key and estimated frequency of flows. For convenience, we use S(i, j) to represent the jth counter in the i array in the sketch, and use B(i, j) to represent the jth counter in the i array in the bloom filter, respectively. Arrays A1,…, Ad are associated with hash functions h1(.),…, hd(.), respectively. These d hash functions h1(.),…, hd(.) need to be pairwise independent. Bloom filter checks whether the min-heap already monitors the flow. For convenience, we use a Boolean variable flag to represent the result of bloom filter. We use H to denote the min heap with k buckets. Let H(t) represent the t bucket in the min heap, and use H(t).key and H(t).value to represent its key field and value field, respectively. Let H(1) represent the root node of min heap, thus H(1).value is the minimal value in min heap. Optimizations In this section, we propose methods to optimize Sketch-BF further to improve speed and reduce error. Optimization 1: aggregative update Problem: If the flow is in the min-heap, then immediately update the min-heap for an incoming packet belonging to the flow is time waste. Solution: Aggregative update. For example, assume flow fi is stored in min-heap, and incoming packets Pl,Pl + 1, Pl + 2 belong to flow fi. Update the sketch to ensure accurate flow frequency, but do not update the relevant value field in min-heap. Only when a new flow fm is not in min-heap and the estimated frequency of fm is larger than the minimum value in min-heap, we update value field of all flows in min-heap and heapify at one time. Let T(n) denote the time of processing an item with frequency of n. Let O(log (k)) represent time complexity for updating min-heap. Assuming that from the end of the min-heap update to the following new flow with frequency exceeding the minimum value of min-heap, the numbers of packets belonging to the flow in the min-heap are, respectively m1,m2,…, mk. Aggregated updates can bring potential benefits of sublinear processing time: T(m1) + … + T(mk) + O(klog (k)) ≪ (m1 + … + mk)(T(1) + O(log (k))) when m1 ≫ 1, …, mk ≫ 1. Optimization 2: Revised bloom filter Problem: assume that there are two flows fi,fj in min-heap share the same counter in bloom filter, then a new flow fl replace fi. Because fi would be deleted in min-heap, the B(i, j) corresponding to fi will set to 0. At this time, an incoming packet belonging to flow fj will be wrong judged that it is not in the min-heap. Solution: Revised bloom filter. First, we use the bloom filter to determine whether the flow exists in the min-heap. If it is judged that the flow does not exist, to avoid false negatives caused by hash collisions, we update the bloom filter counters corresponding to all flows in the min-heap to 1. Then use the revised bloom filter to judge whether the flow exists. Algorithm The insertion and query process of the algorithm is as follows (see pseudo-code in Algorithm 1). ALGORITHM 1. Insertion process for ending top-k flows Insertion: all counters in sketch and bloom filter are initialized to 0. The bucket in min-heap is null. For each incoming packet Pl belonging to flow fi, it will: Step 1: insert fi into d arrays by d hash functions (h1(.),…, hd(.)) in sketch. Sketch-BF will increase the corresponding d counters with 1 and report the minimal counter value n i ̂ $\widehat {{n_i}}$ as the estimated frequency of flow fi. Step 2: get d counters' value in bloom filter with d hash functions which results in Step1. As long as one counters' value is 0, then flow fi is not monitored by the min-heap and set flag to false. Step 3: if flag = false, according to Optimization 1 and Optimization 2, we update the estimated frequency of flows in min-heap and heapify, and revise the bloom filter counter. Step 4: if ∃1 ≤ j ≤ dB (j, h(fi)) = 0, there are two cases for inserting fi into min-heap: (1) the heap is not full, then insert fi into a null bucket in min-heap; (2) the heap is full and n i ̂ $\widehat {{n_i}}$ is larger than the minimal value of min-heap, then set the key and value of H(1) to fi and n i ̂ $\widehat {{n_i}}$ , respectively. Experiment result Experiment setup Dataset The IP Trace dataset we use is anonymized IP traces collected in 2016 by CAIDA [12]. The source IP address identifies each flow in this dataset. We use the first 10 M packets belonging to about 20 K flows. Platform Our experiment runs on a server with four 16-core Intel Xeon Gold 5218 @2.3 GHz CPU and 128 GB DDR4 memory. Implementation We have implemented Sketch-BF, CMH, SpaceSaving (SS), HeavyKeeper, WavingSketch, and DHS in C++. In the experiment, all algorithms use 32-bit to record flow frequency and another 32-bit to record flow key. For Sketch-BF, CMH, and HeavyKeeper, we first allocate memory for the heap including k buckets, and the remaining memory determines the number of counters in sketch. For SS, memory size determines the bucket number, which is usually much larger than k in our experiment. For WavingSketch, the slot number in heavy part is 8 according to the author [4]. Metrics Throughput: we perform the insertions of all packets and calculate the throughput. The throughput is defined as N/T, where N is the total number of packets and T is the total measured time. We use millions of inserted packets per second (Mps) to measure the throughput. Precision: precision refers to the ratio of the correctly reported top-k flows. Advantage of Sketch-BF compared to CMH and HeavyKeeper The critical advantage of Sketch-BF is that for most packets, it does not access the min-heap which keeps the insert speed of Sketch-BF as fast as the conventional sketch. We get the advantage by adding bloom filter variant, which can rapidly check whether the packet belongs to flows in min-heap, instead of slowly traversing heap-part. Another advantage is that for many packets belonging to flows in min-heap, aggregate updates can also significantly improve the processing speed of the data stream. We set the memory size to 20 KB and k to 100 in this experiment. As shown in Figure 2, when the data stream is 10 M, Sketch-BF performs a total of 73,283 heap operations, while CMH performs 7,206,825 times and HeavyKeeper performs 8,650,063. The number of heap operations of Sketch-BF is about 1% of CMH and HeavyKeeper. The result shows that Sketch-BF can significantly reduce heap update operations in min-heap. Fig. 2Open in figure viewerPowerPoint The comparison of operation number Impact of data size on throughput and precision In this section, we measure the effect of data size on throughput and precision. We divide the CAIDA dataset into five different datasets according to the number of packets, the number of packets is 1K, 10K, 100K, 1M, and 10M, respectively. We set the memory size to 20 KB and k to 100. As shown in Figure 3, as data size becomes larger, the throughput of Sketch-BF increases significantly at first, then maintains about 20 Mps. When the data size is 10 M, the throughput of Sketch-BF is 20.5 Mps while it is 3.8 Mps, 5.7 Mps, 4.2 Mps, 10.0 Mps, 4.6 Mps for CMH, HeavyKeeper, WavingSketch, SS, and DHS. The reason is Sketch-BF can greatly reduce heap operations, especially when packet numbers become significant. Fig. 3Open in figure viewerPowerPoint Throughput versus data size Figure 4 shows that the precision is relatively low for all algorithms, and as the data size increases, the precision fluctuates slightly and the overall precision is above 0.92, except that the precision of CMH drops to 0.87 when the data size is 10 M. Fig. 4Open in figure viewerPowerPoint Precision versus data size The experiments show that Sketch-BF achieves the highest throughput and obtains relatively high precision (about 0.95), when the data size is over 10 K. Conclusion We propose a novel solution called Sketch-BF for top-k finding, which is much faster than existing works. The key idea is to significantly reduce the number of update operations in min-heap by adding a bloom filter variant between sketch and heap. Experimental results show that the Sketch-BF can achieve high throughput and precision in top-k finding. We believe Sketch-BF can be applied to many more applications. Acknowledgements This work was supported by National Natural Science Foundation of China: 61872382, 61521003. Conflict of interest The authors declare no conflict of interest. Open Research Data availability statement Research data are not shared. References 1Metwally, A., Agrawal, D., El Abbadi, A.: SpaceSaving: Efficient computation of frequent and top-k elements in data streams. In: Proc. 10th Int. Conf. Database Theory 398– 412 (2005) 2Manku, G.S., Motwani, R.: Approximate frequency counts over data streams. In: Proc. VLDB Endow. 5, 1699– 1699 (2012). https://doi.org/10.14778/2367502.2367508 3Gong, J., Yang, T., Zhang, H., Li, H., Uhlig, S., Chen, S., Uden, L., Li, X.: HeavyKeeper: An accurate algorithm for finding top-k elephant flows. In: Proc. 2018 USENIX Annu. Tech. Conf. USENIX ATC 2018. 909– 921 (2018). https://doi.org/10.1109/tnet.2019.2933868 4Li, J., Li, Z., Xu, Y., Jiang, S., Yang, T., Cui, B., Dai, Y., Zhang, G.: WavingSketch: An unbiased and generic sketch for finding top-k items in data streams. In: Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Min. 1574– 1584 (2020). https://doi.org/10.1145/3394486.3403208 5Wu, M., Huang, H., Sun, Y., Du, Y., Chen, S., Gao, G.: ActiveKeeper : An accurate and efficient algorithm for finding top-k elephant flows. IEEE Commun. Lett. 7798, 1– 5 (2021). https://doi.org/10.1109/LCOMM.2021.3077902 6Cormode, G., Muthukrishnan, S.: An improved data stream summary: The count-min sketch and its applications. J. Algorithms. 55, 58– 75 (2005). https://doi.org/10.1016/j.jalgor.2003.12.001 7Estan, C., Varghese, G.: New directions in traffic measurement and accounting. Comput. Commun. Rev. 32, 75 (2002). https://doi.org/10.1145/510726.510749 8Charikar, M., Chen, K., Farach-Colton, M.: Finding frequent items in data streams. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics). 2380 LNCS, 693– 703 (2002). 9Liu, L., Shen, Y., Yan, Y., Yang, T., Shahzad, M., Cui, B., Xie, G.: SF-Sketch: A two-stage sketch for data streams. IEEE Trans. Parallel Distrib. Syst. 31, 2263– 2276 (2020). https://doi.org/10.1109/TPDS.2020.2987609 10Jie, L., Hongchang, C., Penghao, S., Tao, H., Zhen, Z.: OrderSketch: An unbiased and fast sketch for frequency estimation of data streams. Comput. Networks. 201, 108563 (2021). https://doi.org/10.1016/j.comnet.2021.108563 11Zhao, B., Li, X., Tian, B., Mei, Z., Wu, W.: DHS. In: Proc. 27th ACM SIGKDD Conf. Knowl. Discov. Data Min. 2285– 2293 (2021). https://doi.org/10.1145/3447548.3467353 12 The caida anonymized internet traces 2016, http://www.caida.org/data/overview/ Volume58, Issue11May 2022Pages 429-431 FiguresReferencesRelatedInformation