# SPinDP: A High-Speed Distributed Processing Platform for Sampling and Filtering Data Streams
## 1 Introduction  
This paper introduces an efficient method of applying data purification, a representative technique for data quality management, in a stream environment. A data stream refers to a large amount of data that are continuously generated; most of the data generated online are in stream form. With the recent proliferation of various smart devices and IoT services, data streams are increasing significantly [1, 2, 3, 4]. Unlike conventional bulk data, stream data become very vast as they accumulate, making them even more valuable in the fields of big data and artificial intelligence. In these fields, obtaining high-quality data in various forms, including data streams, is crucial for determining service quality. Therefore, the need for related fields, such as data quality management and data distribution platforms, has also increased. Thus, we present an efficient method of applying data purification to the stream data in the distributed processing environment. Data purification is one of the representative preprocessing steps that apply missing value correction, aggregation, duplicate removal, sampling, filtering, and other techniques to data to improve quality and processing performance. Among them, missing value correction, aggregation, and duplicate removal correspond to data cleaning in structured environments such as RDBMS or NoSQL. In this paper, we focus on designing a purification platform that emphasizes sampling and filtering algorithms that can be continuously applied to data streams, rather than static and simple value corrections. Although many research studies have been conducted on these purification algorithms, there are still severalproblems. First, research on processing systems that can purify large-scale, high-speed, and complex data streams is very limited (Problem 1). Existing stream processing engines mainly focus on fast processing of general streams, but purification systems that focus on handling dynamic streams for actual real-time services have not yet been proposed. Second, there are no specialized systems that provide stream sampling or filtering, so users need to implement and apply the required algorithms themselves for data purification (Problem 2). Moreover, existing purification algorithms are usually based on statistical functions that require data accumulation, making them unsuitable for stream environments with rapid data changes. Third, because users need to implement the necessary algorithms for data purification, it is very difficult for non-experts to apply purification through these processes (Problem 3). Even if applicable algorithms are provided, determining which sampling or filtering to apply to data with different characteristics requires even greater expertise. To address these problems, in this paper we present various sampling and filtering algorithms that can be applied to data streams and propose a new purification platform, called SPinDP (Stream Purifier in Distributed Platform), that can easily utilize these algorithms in a distributed environment. Figure 1 shows the conceptual structure of SPinDP. As shown in the figure, SPinDP consists of four major components: Data Stream Processing Engine, Purification Library, Plan Manager, and Shared Storage. We also utilize open-source solutions for the actual implementation of SPinDP in a distributed environment including Apache Storm [5; 6], the distributed stream processing framework, and Apache Kafka [7; 8], the distributed queuing system. In order to develop the functionalities of SPinDP, we first define the solutions and requirements for existing problems and then design the overall structure and essential functions. Table 1 summarizes the problems and their corresponding solutions, which are implemented by the functions of the components and integrated into the final platform of SPinDP. Figure 1: Conceptual structure with four major components in SPinDP. To clearly connect the solution to each problem, we now describe each component and its role in more detail.