## 2 Related Work and State-Of-The-Art Technologies
### Parallelization and Synchronization  
For maximizing the performance of certain tasks within a workflow, the framework implements a concept for running multiple instances of a workflow task (e.g., a NiFi processor) in parallel. In manyuse cases the results of parallel computations have to be aggregated and synchronized for further use. Therefore, as depicted in Figure 4, the framework provides a coordination service in order to not only allow parallel data processing and execution of a processor but also to fork and join data streams. To describe this concept more mathematically, a dataflow (sort of data stream) can be defined as ordered sequence of data elements:  
\[D_{flow}(i)=V_{i},i\in Z^{+},V\in M, \tag{1}\]  
where:  
\(Z^{+}=\) set of ordered indexes, always positive,  
\(M=\) set of possible (structured) data values. A common form of parallel dataflow processing would now be to distribute the elements of a dataflow to a set of parallel instances of a workflow task such that each instance of the task carries out the same computation on different elements of the dataflow concurrently (e.g., fork the dataflow). Even in this scenario, many solutions for distributing the data elements to the different processor instances are possible, so let us take a simple \"worker pool\"-like distribution where each data element is scheduled to be processed by the next free task instance. For implementing such a dataflow processing strategy in the presented framework, first the number of concurrent tasks and the type of distribution (e.g., \"worker pool\") has to be defined in the config interface of a NiFi processor by the workflow editor. Then, an executable coordinator, which is located in each NiFi processor as part of the abstract processor class, will launch the given number of parallel working instances in a corresponding number of separate containers. Additionally, the \"work pool\"-based distribution of the dataflow elements will  
Figure 4: Example of the parallelization and coordination concept with unsorted result data. be instrumented in the communication layer. Therefore, as shown in Figure 4, a dataflow \(D_{flow}\) is transferred to the executable coordinator and stored in a queue for distribution. By using the different communication channels and the distribution strategy, the executable coordinator distributes data elements to the task instances, e.g., \(D_{flow}(1)\), \(D_{flow}(2)\) and \(D_{flow}(3)\) are handled in parallel by the task instances in the containers x, y and z. Once one instance finishes its data processing, the corresponding result is transmitted back to the coordinator, and the instance will obtain the next input data from the queue, e.g., as presented in Figure 4, after the result data \(R_{1}\) and \(R_{3}\) arrives at the coordinator, the next input data \(D_{flow}(4)\) and \(D_{flow}(5)\) are transferred to the two available instances in containers x and y. If there are no further processing requirements for the sequence of output data, the coordinator will pass the received result data to the next NiFi processor (this strategy is depicted in Figure 4). The results will be collected into an output data flow (join operation) which is normally sorted based on availability of the results. If the result data have also to be sorted in the same order as the input dataflow defined by the workflow editor (\"keep order\"), the coordinator will sort the result data sequentially based on the index \(i\) of the input elements. In addition, if for one dataflow, the NiFi processor getting the output dataflow requires that all parallel processed result data must be available as one large result data set, the coordinator will wait until all result data arrive completely, then sort and join them together as one large result data set according to the indices \(i\), and finally transfer it.