# SPinDP: A High-Speed Distributed Processing Platform for Sampling and Filtering Data Streams
## 4 Experimental Evaluation
### Evaluation of Purification Functionality  
The functionality evaluation of SPinDP is divided into sampling and filtering. First, Figure 9 shows the results of the original data and the samples extracted by each sampling algorithm. In the figure, the top left is the original Apple finance data, and the rest are the results of the samples extracted by each sampling algorithm. Through these graphs, we can intuitively observe that the trends and shapes of the original and the samples are very similar. This means that the sampling algorithms supported by SPinDP work correctly in purifying the real data stream. The evaluation of filtering functionality is verified through query filtering, which involves refining values below 290,000 KRW from the Ethereum market stream. Figure 10 shows the query filtering results on a day with different patterns from the daily 1-min price data. In the figure, the blue line is the original stream, and the orange line is the filtered stream. Based on the experimental results, we can see that the query filtering accurately refines the values below 290,000 KRW. In other words, values in the source data that are less than the threshold of 29,000 won will be removed by query filtering. This result means that query filtering, which is a simple but precise method to obtain only the data users want, works correctly in the stream environment built by SPinDP. Figure 11 shows the data processing throughput (i.e., tuple processing throughput) measured during the execution of the purification algorithms. As shown in the graph, we can see that all purification algorithms have higher throughput in the order of Ethernet, IPoIB, and RDMA. In particular, considering that the vertical axis of the graph is logarithmic, we can observe that there is a significant difference in performance between different network environments. As a result, in the throughput comparison experiment, we confirmed that the performance of RDMA improved by an average of 12.6 times compared to IPoIB and up to 28.0 times compared to Ethernet. In the case of Bloom filtering, the throughput of RDMA was improved by up to 27.6 times compared to IPoIB and 66.5 times compared to Ethernet. This is because the complexity of the Bloom filtering Topology increased compared to the other purification algorithms, leading to an increase in the number of tuple transmissions between tasks. As the number of tuple transmissions between tasks increases, the external network usage also increases, ultimately resulting in different overall throughput depending on the network environment. Figure 10: Query filtering results of cryptocurrency market stream. Figure 9: Apple finance data sampling results for each algorithm. The results of the latency experiments, shown in Figure 12, also demonstrate that RDMA is faster than both IPoIB and Ethernet. As shown in the figure, the latency of RDMA is 2819.4 times faster than IPoIB on average and 2473.7 times faster than Ethernet. This is a much larger difference than the performance improvement seen in the tuple processing throughput, and there are two major reasons for this. First, InfiniBand has an around 60.0 times bandwidth advantage compared to Ethernet, and in the RDMA-Storm environment, locality-aware dynamic grouping is adopted to minimize communication with other nodes and to maximize processing efficiency. Moreover, even when communication with other nodes is required, RDMA can read and process tuples directly from memory without going through the operating system, which likely contributed to the reduction in latency. Second, bottlenecks are inevitable in Topology structures, but with RDMA and locality-aware dynamic grouping, such bottlenecks can be addressed more efficiently. In addition, since the input stream is continuously incoming without latency, the latency in processing can accumulate in the event of a bottleneck, resulting in the performance differences observed in this experiment. In the latency experiment results, all algorithms except for Bloom filtering show that IPoIB is slower than Ethernet.