# SPinDP: A High-Speed Distributed Processing Platform for Sampling and Filtering Data Streams
## 1 Introduction
The first component, Data Stream Processing Engine, performs the distributed processing of data streams through purification algorithms and coordinates the entire system through the Plan Manager and Shared Storage. To solve Problem 1, Data Stream Processing Engine aims to provide purification algorithms that supports large-scale, high-speed data stream processing. In this paper, we design this component based on RDMA-Storm [9], which is accelerated by InfiniBand RDMA (Remote Direct Memory Access) [9], and Apache Kafka [7,8] for seamless stream I/O and transmission management. The second component, Purification Library, is implemented in Storm's topology structure, and SPinDP provides a total of eight sampling and five filtering algorithms. In particular, three of these algorithms are improved for suitability in data streams through acceleration/intelligence, and the other ten algorithms also have been optimized for distributed stream processing environments using shared memory, etc. With the stream-specific purification algorithms provided by SPinDP, users can easily utilize more diverse purification methods, which solves Problems 2 and 3. The third component, Plan Manager, is a Web-based client that enables efficient use of SPinDP. The fourth component, Shared Storage, performs joint data storage and management among Plan Manager, Data Stream Processing Engine, and Purification Library. Based on these two components, SPinDP provides essential functions for platform users and administrators, such as defining data stream input and output methods, defining data types and schemas, creating/modifying/executing/deleting purification plans, and monitoring execution plans. Plan Manager and Shared Storage improve the accessibility and utilization of stream purification algorithms, which addresses Problem 3. It can also be seen as an indirect solution to Problem 2, as Plan Manager makes it easy for users to drag and drop the suitable combination of purification algorithms for their own data. SPinDP can be applied in various fields, with representative use cases as follows:  
* Smart city technology: Utilized for refining smart city data collected from various sensors and IoT devices, providing precise insights into traffic, energy, the environment, and more. * Cyber security: It is important to purify stream data to quickly detect anomalies in network traffic and protect against attacks. * Medical monitoring: Stream purification is needed to monitor patients' vital signs in real-time and remove noise to help make accurate medical decisions. * Neuroscience research: Removing noise from EEG or fMRI data, which measures brain signals, allows for a more accurate analysis of brain activity. * Environmental surveillance: Environmental data from sensor networks is refined and utilized to derive accurate information about air quality, water quality, and more. * Financial trading analysis: In the stock market, sampling and filtering are used to remove noise from trading data and identify accurate trading signals. \begin{table}
\begin{tabular}{p{142.3 pt} p{142.3 pt}} \hline \hline
**Related Problems** & **Solutions** \\ \hline Problem 1 & (Data Stream Processing Engine) Design a new high-speed distributed parallel processing engine based on ultra-fast networking technology \\ \hline Problems 2 and 3 & (Purification Library) Development of stream sampling and filtering algorithms for supporting multiple and complex data streams \\ \hline Problem 3 (partially Problem 2) & (Plan Manager \& Shared Storage) Support for Web-based user interface tools for both expert and non-expert users \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The problems and their corresponding solutions with SPinDP components. Manufacturing and quality control: Refining and analyzing sensor data from the manufacturing process can help you accurately monitor quality and improve production line efficiency. As you can see from these examples, stream purification is essential for real-time analytics.