# Application of Workflow Technology for Big Data Analysis Service
## 2 Big Data Analysis Oriented Cloud Workflow System
### Workflow Execution Engines
#### 2.2.2 Online Real-Time Analysis Model  
The MapReduce architecture focuses on batch computing analysis, which can satisfy many offline analysis needs. However, many highly dynamic real-time analyses (e.g., real-time trend prediction and search query) are needed. We employ Storm [15] in the distributed computing platform to analyze the sequential streaming data.  
Storm is different from MapReduce. MapReduce is essentially a batch analysis system which puts data onto the HDFS and distributes them to each node to be analyzed. After each node completes data analysis, the resulting data are sent to HDFS for users. Storm is in-memory computing, which imports data onto memory for computing. Reading from and writing to memory are faster than hardware conditions because the hardware access latency is approximately 75,000 times higher than the memory access, indicating that Storm is better suited to real-time computing than MapReduce.  
As shown in Figure 9, Storm implements a data streaming model which provides a series of basic elements for parallel computing, such as topology, stream, spout, and bolt. Storm encapsulates a real-time task, given topology includes spouts and bolts, and forms an execution graph based on data streaming. The topology is submitted to a cluster and the master node distributes tasks to worker nodes for execution.  
Figure 8: ChainWF mechanism.  
The Storm cluster distributes needed resources for the topology to ensure its smooth implementation. When the topology is completely executed, it releases the occupied computing resources to the Storm cluster. An online real-time workflow can be expressed using pseudo-regular expressions as follows:  
\[\left[\text{Spout}\,|\,\text{Bolt+}\right]+\]  
Topology supports one or more tasks to complete one computing component (Spout or Bolt) in parallel, and the Storm cluster distributes multiple tasks to run the component on the basis of users' definition. Storm defines a stream grouping strategy to transfer tuples between spout and bolt. In topology construction, users need to define which stream type can input data onto the bolt. Storm currently provides seven stream grouping strategies: shuffle grouping, fields grouping, all grouping, global grouping, non grouping, direct grouping, and local and shuffle grouping.  
Figure 10 shows that the Storm cluster has two nodes--namely, master and work nodes.  
* The master node runs the daemon Nimbus, which is similar to JobTracker in Hadoop. Nimbus is mainly responsible for distributing code, allocating tasks to the work node, and monitoring the running state of the cluster.
* The work node runs the daemon Supervisor which monitors tasks allocated by Nimbus and start/stop worker process. Each worker process executes the topology sub-tasks. Multiple distributed work nodes can cooperate to complete a Topology.  
Zookeeper can coordinate the interaction between Nimbus and Supervisor and store all the running states in the Storm cluster. The processes of Nimbus and Supervisor are fail-fast or stateless.  
Figure 10: Real-time workflow tasks executive process. YARN: Yet Another Resource Negotiator.  
Figure 9: Storm topology.