# A Data Processing and Distribution System Based on Apache NiFi
## 5 Research and Results  
As the environment presented in Section 4 serves commercial and production purposes, we were not able to utilize it to conduct our research. Furthermore, we also had to use other datasets to keep the original data confidential. However, despite these differences, our work can be used directly in the context of monitoring optical networks in 5G/6G deployments. All experiments were carried out on the CORD-19 dataset ([[https://www.kaggle.com/datasets/allen-institute-for-ai/CORD-19-research-challenge?select=document_pares](https://www.kaggle.com/datasets/allen-institute-for-ai/CORD-19-research-challenge?select=document_pares)]([https://www.kaggle.com/datasets/allen-institute-for-ai/CORD-19-research-challenge?select=document_pares](https://www.kaggle.com/datasets/allen-institute-for-ai/CORD-19-research-challenge?select=document_pares)), accessed on 13 February 2023). We investigated the effect of the number of nodes in a cluster on the performance of Apache NiFi. The task carried out by the Apache NiFi instance under study was that of extracting a unique list of scientific papers cited in the articles included in CORD-19. Among other things, the title, authors, and publication date were extracted. The input data were in JSON format, and the results were saved as csv files. This experimental dataset can be easily used to benchmark analogous operations in the context of optical network monitoring in 5G production environments. The only difference will be changing the titles and dates of the research papers into the selected parameters extracted from the log files collected from optical devices. Thus, the particular content of each file is not critical. We chose this particular dataset to perform our experiments because it is easily available at no charge and is well documented. Regardless of the data context, the processing path was divided into three parts, as shown in Figure 7. The first loaded the data, the second performed the actual processing, and the last saved the results. Detailed documentation and configuration files, the configuration of each individual FlowFile Processor, and the connections between them were placed in a git repository ([[https://github.com/karolwn/Apache_NiFi](https://github.com/karolwn/Apache_NiFi)]([https://github.com/karolwn/Apache_NiFi](https://github.com/karolwn/Apache_NiFi)), accessed on 13 February 2023). Figure 8 presents the logical scheme of the experiment, which was as follows. The downloaded CORD-19 dataset was trimmed to 200,000 JSON files (17.7 GB). This was done to reduce the required disk storage space. This subset was later compressed, and before \begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Physical Device** & **Optical Port** & **Trail** \\ \hline Kafka (input) & 95,894 & 1,536,536 & 291,796 \\ Reference DB (output) & 1935 & 49,524 & 13,945 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of the data processing. Figure 7: Configured processing path. each iteration of the experiment, it was extracted to the shared input folder. Apache NiFi read data from this folder and processed them, as shown in Figure 7. The obtained results were saved in a different shared folder to prevent input and output data from mixing. In the final step, performance statistics were calculated based on the processed entries and additional information about the utilization of virtual machines (the script is available in the git repository ([[https://github.com/karolwn/Apache_NiFi](https://github.com/karolwn/Apache_NiFi)]([https://github.com/karolwn/Apache_NiFi](https://github.com/karolwn/Apache_NiFi)), accessed on 13 February 2023)). In this work, we propose an original modification of the Apache NiFi configuration to improve the data processing speed. To achieve faster processing, we first suggest modifying the default values of the Concurrent Tasks and Run Duration parameters that are equal to 1 and 0 ms, respectively. Table 2 presents the proposed values for consecutive processors according to the order in the processing path.