# SPinDP: A High-Speed Distributed Processing Platform for Sampling and Filtering Data Streams
## 3 SPinDP: Stream Purifier in Distributed Platform
### Data Stream Processing Engine
Based on the newly improved class structure, TCP/IP (Netty) or RDMA communication can be set in the actual Storm configuration, and data transmission between workers is performed through each class according to the corresponding setting. The SPinDP proposed in this paper also utilizes this structure of RDMA-Storm in its Purification Library. Process-grouping-based performance enhancement: Apache Storm generally uses grouping techniques to determine which node to send data to during stream processing. Storm provides various grouping methods, and our SPinDP introduces locality-aware dynamic grouping [25] to address the issues of shuffle grouping and static grouping. Shuffle grouping randomly determines the object to receive and send data, but it does not consider the physical distance between nodes or the network conditions. Therefore, network latency can be long even when distributing the same amount of data. To alleviate this issue, Storm supports local-shuffle grouping, which minimizes the latency by processing data on the same node when the object to receive the data is on the same node during shuffle grouping. However, local-shuffle grouping only considers locality, which causes the problem of object transmission imbalance. For example, if the number of Spouts (sending objects) per node is different, the Bolts (receiving objects) on nodes with no sending objects become idle as they have no data to process. On the other hand, if the number of receiving objects is imbalanced among nodes with the same number of sending objects, the nodes with fewer objects may incur severe overload. To address this grouping issue, the proposed SPinDP utilizes locality-aware dynamic grouping [25]. This calculates the distance between each node through ping operations, sets weights based on the calculated distances, and determines the nodes to receive data based on the set weights. The weight \(Closeness(N_{i},N_{j})\) is calculated using Equation (1), where \(n\) is the number of nodes, \(N_{i}\) is each node, \(Task(N_{i})\) is the number of tasks (objects) on node \(N_{i}\), and \(Ping(N_{i},N_{j})\) represents the ping time from node \(N_{i}\) to \(N_{j}\). In the end, the more objects there are on a node and the shorter the ping response time, the greater the weight assigned to that node [25]. \[ \begin{split}& Close(N_{i},N_{j})=\frac{Task(N_{j})}{Ping(N_{i},N_{ j})\times TotalPing(N_{i})},\\ &\text{where }TotalPing(N_{i})=\sum_{j=1}^{n}\frac{Task(N_{j})}{Ping(N_{ i},N_{j})}\end{split}  \tag{1}\]  
However, if grouping is based only on weights, it is difficult to solve the load balancing problem caused by the sender-receiver object imbalance. This is because selecting a receiving node based only on weight can lead to traffic congestion in certain nodes, causing another problem. Therefore, in the final node selection, we use a random number along with the calculated weights to probabilistically select other nodes appropriately, distributing traffic to prevent idle or overloaded states in specific nodes. As a result, with the locality-aware dynamic grouping supported by SPinDP, more streams can be sent to nearby nodes to minimize the processing latency, and stable stream processing can be provided by distributing traffic probabilistically to prevent overload in some objects. The newly implemented Data Stream Processing Engine, which has been improved through infrastructure-based and process-grouping-based performance enhancement, is utilized as a core technology of the proposed platform along with Apache Kafka. The Purification Library, Plan Manager, and Shared Storage, which will be explained later, also operate around this Data Stream Processing Engine, and their performance will be evaluated in Section 4.