# SPinDP: A High-Speed Distributed Processing Platform for Sampling and Filtering Data Streams
## 4 Experimental Evaluation  
### Experimental Methods and Environment  
In this section, we conduct an experimental evaluation to verify the functionality and efficiency of the implemented SPinDP platform. SPinDP is designed primarily for a distributed cluster environment, as shown in Figure 7. For the experimental data, we used virtual stream data consisting of randomly generated numbers (integers and real numbers) and text (1 KB size array) that were collected and generated internally for SPinDP platform operations. Figure 6: Overall architecture of Plan Manager. Figure 7: Cluster diagram of the SPinDP. The hardware and software environments used in our experiments are shown in Table 2. As shown in the table, a distributed cluster consisting of one master node and eight worker nodes is the core hardware environment of SPinDP. The distributed cluster consists of a main system based on Apache Storm and Apache Kafka, and a Web server for Plan Manager. We also built a separate deep learning environment for training the models used by the intelligent algorithms in Purification Library, and the inference models trained on that server run in the same distributed environment as the main system. To evaluate the functionality of SPinDP, we verify whether the sampling and filtering algorithms supported by Purification Library effectively refine real data. The criteria for measuring the purification results of sampling and filtering are very different depending on the application or environment where the purified data will be used. Thus, in this paper, we evaluate the proper operation of sampling in terms of whether it reflects the pattern of the original data, and filtering in terms of whether it purifies the data according to the conditions. Each experiment utilizes Plan Manager, and Figure 8 provides an example screen of the Web interface used in the experiment. For sampling evaluation, we use reservoir, priority, hash, systematic, KSample, UC KSample, and binary Bernoulli sampling, and represent the original and sampled streams graphically to see whether each algorithm captures the patterns in the original. As for filtering, since each algorithm has different conditions and objectives, it is more challenging to determine the effectiveness of the purification; therefore, we evaluate query filtering, which allows the purification results to be intuitively verified. The original data for each experiment consist of Apple finance data (2010-2018) and cryptocurrency market data (July 2017). Apple finance data consists of daily closing prices over a period of 9 years, comprising approximately 3280 tuples. Additionally, for cryptocurrency market data, we utilized six sets of 1440 tuples each, representing the per-second prices of Bitcoin measured throughout a day. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Component** & **Software** & **Hardware** \\ \hline \multirow{4}{*}{Data Stream Processing Engine} & \(\bullet\)**OS:**CenROS 7 & \\  & **Stream processing:** Apache Storm 2.0.0 & \\  & **Queueing:** Apache Kafka 2.1.1 & \\  & **Coordinator:** Apache Zookeeper 3.4.10 & \(\bullet\)**Master (1 node): Intel E5-2630V3** \\ \hline \multirow{2}{*}{Purification Library} & \(\bullet\)**RDMA library:** JXIO 1.0.3 & 2.4 GHz 8 Core, 32 GB RAM, \\  & \(\bullet\)**RDBMS:** MariaD 10.0 & 256 GB SSD, 1 TB HDD \\ \hline \multirow{4}{*}{Shared Storage} & \(\bullet\)**In-memory DB:** Redis 5.0 & \(\bullet\)**Worker (8 nodes): Intel E5-2620V3** \\  & \(\bullet\)**Server:** Apache Tomcat 9.0 & 256 GB SSD, 1 TB HDD \\ \hline \multirow{4}{*}{Plan Manager} & \(\bullet\)**Client:** Java 8.0 & \(\bullet\)**HefininBand:** Mellanox SwitchX®-3 MSX \\  & **Language:** SP: HTML, CSS, JavaScript & 6012F-1 BMS Managed FDR56 Gbps \\ \cline{1-1}  & **Libraries:** Ajax, \(\{\)Query, j\#Plump, Query-Builder & \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Hardware and software specifications for core components. Figure 8: An example screen of Plan Manager.