# Application of Workflow Technology for Big Data Analysis Service
## 2 Big Data Analysis Oriented Cloud Workflow System
### Workflow Execution Engines
#### 2.2.1. Offline Batch Analysis Model  
The offline batch analysis model employs a Hadoop job to execute offline analysis tasks. Each job involves Map and Reduce stages, but the Reduce stage is unnecessary. Figure 7 shows the process of job execution in a cloud platform. When the daemon submits a task, JobTracker coordinates resources to execute it first (M1, M2, and M3) at the Map stage and then at the Reduce stage (R1 and R2). The number of Map and Reduce tasks can be configured by users (3 and 2 in Figure 7 respectively). A pseudo-regular expression is used to demonstrate the job execution: [MAP! REDUCE?]. TaskTracker monitors the entire job execution at the Map and Reduce stages, which run on Java virtual machines  
Figure 6: Batch workflow execution process.  
independently. The job inputs and outputs in the HDFS, as well as the input data will be sent to a particular location in the HDFS.  
The daemon decomposes the DAG into the run time dependencies among jobs. Nonlinear dependencies can be implemented by modular MapReduce. Hadoop provides job and JobControl classes. The former maintains the configuration of sub-tasks and the dependencies among sub-tasks, and the latter controls the entire task process by adding all tasks into JobControl and executing the run() function to ensure the sequential execution of workflows. Under common workflow condition, tasks are executed dependently and sequentially. The definition is as follows:  
**Definition 1**.: _CommonWF: Common workflow can be described using pseudo-regular expression as follows:_  
[MAP | REDUCE?] +  
_REDUCE is represented as the reducer stage following MAP which means mapper. [\(\cdot\)] denotes a job and [MAP | REDUCE?] means to repeat the process one or more times to complete tasks._  
Chain WorkFlow  
In the case of a common workflow, the output results of each component are written into the hardware. For component-intensive workflows, frequency I/O operations will decrease the workflow performances. In an actual data analysis process, massive data analysis applications need to preprocess and postprocess the data (e.g., select, where, and casewhen). These preprocess or postprocess components are implemented by the MapReduce task in the case of common workflows.  
Figure 7: Batch workflow tasks executive process.  
Inefficiency occurs when the intermediate results of each step take up I/O and storage resources. The preprocess and postprocess components are often simple and only have mapper functions. Therefore, the workflows can be executed in a modular or combined way. If multiple jobs only have one reducer, they can be combined as ChainJob, which can execute together in a memory (as shown in Figure 8). The combination of multiple process components does not generate the intermediate file or reduce the I/O operations to any great degree. On the basis of the preceding analysis, we propose a chain workflow mechanism wherein the daemon combines jobs dynamically. The definition is as follows:  
**Definition 2**.: _ChainWF: The pseudo-regular expression of ChainWF is:_  
MAP +! REDUCE! MAP*  
With ChainJob2 in Figure 8, for example, the ChainWF engine executes multiple mapper preprocess functions with reduce function following, then optionally executes some mapper for postprocessing. In practice, ChainWF combines multiple tasks into a ChainJob and executes them together, thereby improving the performance of workflow execution.