# Application of Workflow Technology for Big Data Analysis Service
## 1 Introduction  
With the rapid growth of data types and scale, the requirements for data analysis capabilities have been further enhanced. A data processing system (DPS) [1] collects and manipulates data items to produce meaningful information. Data analysis can assist companies in making market decisions, supporting business development, and improving user experience. Traditional data analysis systems are mostly stand-alone, such as SPSS ([[https://www.ibm.com/analytics/data-science/predictive-analytics/spss-statistical-software](https://www.ibm.com/analytics/data-science/predictive-analytics/spss-statistical-software)]([https://www.ibm.com/analytics/data-science/predictive-analytics/spss-statistical-software](https://www.ibm.com/analytics/data-science/predictive-analytics/spss-statistical-software))), Statistical Analysis System ([[https://www.sas.com/en_us/home.html](https://www.sas.com/en_us/home.html)]([https://www.sas.com/en_us/home.html](https://www.sas.com/en_us/home.html))), R ([[https://www.r-project.org/](https://www.r-project.org/)]([https://www.r-project.org/](https://www.r-project.org/))), and Waikato Environment for Knowledge Analysis ([[https://www.cs.waikato.ac.nz/ml/weka/](https://www.cs.waikato.ac.nz/ml/weka/)]([https://www.cs.waikato.ac.nz/ml/weka/](https://www.cs.waikato.ac.nz/ml/weka/))). In the era of big data, data processing systems are mostly built on the cloud platform and employ distributed computing technology. The main data analysis methods, such as extract transform load (ETL) process, data mining, querying, and online analytical processing (OLAP), have also been implemented in a distributed computing manner to handle massive data analysis. This study focuses on how to introduce the workflow technology into the big data analysis system so as to improve the analysis performance. With increasing information, traditional simple data processing systems meet the following challenges: (1) tremendous amounts of data: the scale of data to be processed becomes increasinglylarge; (2) data complexity: many new data types (e.g., structured and semi-structured data) emerge; (3) diversity of mixed loads: numerous emergent self-analysis requirements are proposed with the development of enterprises and the diversity of data. Cloud computing [2; 3] provides a new solution to traditional data analysis problems. It is a style of computing in which dynamically scalable and often virtualized resources are provided as services over the Internet. This computing style allocates computing tasks to parallel computers rather than local computers or remote servers. The operation of the enterprise data center will be similar to the Internet, which promotes enterprises to allocate limited computing and storage resources to required applications. Cloud computing provides services such as infrastructure-as-a-service (IaaS), platform-as-a-service (PaaS), and software-as-a-service (SaaS). Large amounts of data also promote the development of distributed computing technologies. Many studies have been conducted on the batch computing of big data [4; 5; 6]. Google proposed the MapReduce programming model, and its open-source Hadoop has obtained remarkable achievements in theory and practice [7]. Not only batch computing, but also real-time streaming data analysis has drawn the attention of enterprises. For example, Yahoo introduced the S4 distributed stream computing system, and Twitter developed the Storm streaming computing system. Facebook utilizes Data Freeway and Puma to manage real-time data, whereas Microsoft uses the TimeStream system. To meet these challenges, we propose a distributed big data analysis system--namely, distributed data mining (DDM) [8; 9], which is a typical SaaS. Figure 1 shows the DDM architecture which contains infrastructure, algorithm, workflow, application, and system management. DDM can analyze and store large amounts of data on the basis of parallel computing and distributed storage capacity.