# SPinDP: A High-Speed Distributed Processing Platform for Sampling and Filtering Data Streams
## 3 SPinDP: Stream Purifier in Distributed Platform
### Data Stream Processing Engine  
The first component, Data Stream Processing Engine (Data Stream Processing Engine is available at [[https://github.com/dke-knu/i2](https://github.com/dke-knu/i2) am/tree/master/rdma-based-storm]([https://github.com/dke-knu/i2](https://github.com/dke-knu/i2) am/tree/master/rdma-based-storm) (accessed on 1 December 2023)), is a distributed framework based on open-source software that is responsible for the actual execution of various algorithms for purifying input streams. The Data Stream Processing Engine works in three stages: stream input, purification, and output. In SPinDP, Apache Kafka is utilized for stream input and output, while Apache Storm is used for stream purification. Most existing stream processing engines execute all three stages in a single program unit, which is highly inefficient as it requires stopping the running program, manually modifying the code, and restarting it when changes are made to input or output information. Additionally, if there is no intermediate storage, streams that are input during the service interruption may be lost. Therefore, in SPinDP, Apache Kafka manages stream input and output, allowing Apache Storm to focus more on the  
Figure 4: Overall architecture of SPinDP. stream purification itself. To integrate Kafka and Storm, the KafkaSpout class provided by Storm is used as input, and the final output Bolt is configured to store results in Kafka. In particular, by registering the Kafka producer keys with Storm's Spouts, new sources can be seamlessly integrated with the producer even if the input source changes, thus providing uninterrupted service. Apache Storm, which is the basis of Data Stream Processing Engine, is already a well-made framework designed for stream environments. However, its existing structure has limitations in handling fast-paced big data streams generated from multiple input sources. Particularly, as the format of data streams has expanded from simple text to multimedia and their generation speed has become increasingly fast, there is a need for an advanced stream processing platform that can operate robustly in such changing environments. Based on this observation, in this paper we propose a redesign of Apache Storm by applying two performance enhancement approaches, infrastructure-based and process-grouping-based, to create a stream processing engine that can be utilized for the proposed SPinDP. Infrastructure-based performance enhancement: In the existing Storm, the Netty library is used for asynchronous input/output communication between nodes via TCP/IP. The threads responsible for sending and receiving data between Storm workers repeat the process of being activated by data transmission based on Netty and then waiting after the completion of data transmission. Context switching of threads that occur during this execution-waiting process is carried out through the CPU, and, accordingly, the CPU overhead increases as the amount of communication between workers increases. Additionally, during the process of transmitting data based on TCP/IP, frequent buffer copying within the operating system consumes a significant amount of CPU resources. To address this issue, SPinDP introduces RDMA-Storm [9], which applies InfiniBand-based RDMA technology to network communication in Apache Storm, improving the performance of the proposed Data Stream Processing Engine. RDMA-Storm is a new framework that combines Apache Storm with an efficient method of directly accessing memory on other nodes without burdening the CPU by transmitting data in InfiniBand environments. By using RDMA, the network adapter can read and write data directly from memory, thereby eliminating the buffer copying process required by TCP/IP. This solution resolves the issue of CPU overhead and can also improve the overall communication performance of the entire cluster. Figure 5 represents the final class structure of RDMA-Storm. As shown in the figure, RDMA-Storm modifies and extends the legacy classes to use both Netty- and RDMA-based communications.