# A Data Processing and Distribution System Based on Apache NiFi
## 5 Research and Results
### One Node  
Figure 9 shows the variation in the load average parameter value during the data processing. The different line styles denote results averaged from the last one, five, and fifteen minutes. The vertical dashed unlabeled line in all graphs indicates when the optimized node finished processing the input set against the default configuration. The load average values of the optimized node oscillated around a value of 8.5, while for the default case, the corresponding value was equal to 7. This difference resulted from the higher average CPU usage of the node for the configuration proposed in this article. The fluctuations in the one-minute load average were caused by the scheduler settings of the individual FlowFile Processors because they did not run continuously. Instead, they were triggered by the scheduler. Wide confidence intervals in the case of the default configuration demonstrated the unstable operation of the infrastructure. For the default configuration, the values of the load average for even a 15-min period ranged from 4 to 9 between the 60 th and 160 th minutes of the experiment. This discrepancy resulted from the completely full queues, which caused interruptions in the runtime of the FlowFile \begin{table}
\begin{tabular}{c c c} \hline \hline
**Processor** & **Concurrent Tasks** & **Run Duration [ms]** \\ \hline GetFile & 10 & N/A \\ NiFiFlow & 10 & N/A \\ EvaluateJsonPath & 10 & 500 \\ UpdateAttribute & 10 & 500 \\ JolfTransform]SON & 7 & 1000 \\ EvaluateJsonPath & 2 & 1000 \\ Splitlyson & 3 & 500 \\ EvaluateJsonPath & 10 & 2000 \\ ExtractAuthors & 5 & 2000 \\ UpdateAttribute & 5 & 2000 \\ ReplaceText & 10 & 2000 \\ HashContent & 5 & 2000 \\ DetectDuplicate & 12 & 2000 \\ ReplaceText & 10 & 2000 \\ MergeContent & 5 & N/A \\ ReplaceText & 5 & 50 \\ UpdateAttribute & 1 & 0 \\ PutFile & 5 & 25 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Proposed parameter values for individual FlowFile Processors. Processors. The proposed optimization eliminated these problems and ensured the correct and stable operation of the system. Thus, the proposed improvements enabled the use of Apache NiFi in 5 Gproduction networks by ensuring its reliability under demanding conditions. In the future, an analogous solution will also be applicable to 6G infrastructure. The individual residence times of the FlowFile in the system, i.e., the times needed to extract a single resulting entry, are shown in Figure 10. The proposed optimization reduced the average processing times by approximately one order of magnitude from 1406 to 102 s. This means that the state of the 5G infrastructure reported by the optical network devices could be processed up to ten times faster than in the default configuration. It further significantly improved the effectiveness of monitoring of the complex environment. For example, information about failures in the optical layer of the 5G/6G network will be delivered to the NOC (network operations center) 10 times faster. This could significantly reduce the impact of any events and, thus, the potential costs for the operators of 5G and future 6G networks. Additionally, the response time for any failure could be shortened, which could positively impact the overall quality of service (QoS) offered to the client. Table 3 presents the utilization of CPU and RAM as a function of the cluster size for both the default and optimized configurations. The Apache NiFi instance that was  
Figure 10: Distribution of time required to extract a single cited article (one node). Panel (**a**) shows the result of the optimized configuration, whereas (**b**) depicts the default configuration. Figure 9: Load average parameter as a function of the processing time of the input set (one node). Panel (**a**) shows the result of the optimized configuration, whereas (**b**) depicts the default configuration. subjected to optimization used the server's CPU resources approximately 30 percentage points more efficiently compared to the default configuration when using a single node.