# A Data Processing and Distribution System Based on Apache NiFi
## 4 Commercial Deployment of NiFi to Monitor a 5G Optical Network  
In this section, we present how Apache NiFi supports the monitoring of optical infrastructure in a 5G network. We focus on a particular deployment as an example. However, an analogous solution can be applied to 6G infrastructure. Figure 1 illustrates the production environment. Statistics from optical transceivers are collected and aggregated to different Kafka topics by the client's internal solution. These steps could be easily incorporated into the NiFi. The Apache NiFi block represents the entire data processing flow. In this work, we focus on the performance of this part. A reference database (DB) stores the results. Each row consists of such information as an identifier, model, serial number, or last update date. These data are later used to deliver services requested by the client--for example, a graphical representation of the optical infrastructure in the form of a tree hierarchy. As a whole, the delivered solution applies to multiple 3 GPP and tmforum specifications--for example, those for management and orchestration [22; 23]. The data model is based on [24]. The first task performed by NiFi was that of retrieving data from various Kafka ([[https://kafka.apache.org/](https://kafka.apache.org/)]([https://kafka.apache.org/](https://kafka.apache.org/)), accessed on 13 February 2023) topics that were aggregating information about different elements in the optical network, i.e., devices and transceivers. A dedicated topic was assumed for each network interface card. Due to the high configurability, NiFi was able to process all of the data without the need for additional dedicated services. Instead, the information from heterogeneous optical devices was dynamically routed to different process groups, which handled object types that were specific to particular devices and vendors. As a result, the platform can be easily modified and extended. It is also resilient to processing errors, as flows are independent of each other. NiFi is also capable of performing the active approach that we investigated in our work. Namely, the SNMP can be used to automatically receive logging data from optical devices in a 5G network (represented as datasets in our work). Part of our solution can be viewed in Figure 2, which presents the system components responsible for loading the data. Figure 1: Architecture of the commercial solution. In the next step, NiFi extracted some fundamental information about fetched objects--for example, the identifier and the type of requested operation (e.g., create, update, or delete). Then, FlowFiles were processed by the modules to provide a mapping between the JSON format (data loaded from Kafka) and an SQL statement. Figure 3 presents the mapping process. Each processor group (grouped FlowFile Processors; here, they are labeled _Prepare... query_) was dedicated to a different object type. Based on the value of the _type_ field, the _RouteOnAttribute_ FlowFile Processor decided about passing a FlowFile to the proper mapping block. Each process group consisted of three main parts. The first one prepared DELETE statements, the second logged processing errors, and the final part (presented in Figure 4) handled the CREATE/UPDATE requests (these types came from the architecture of the whole commercial system; their names refer to an operation that must be performed on an object specified in a message). Figure 4 presents the universal logic because each object type had a similar CREATE/UPDATE part that consisted of two main components. First, the _ExtractAt  
Figure 4: Fragment of a mapping block dedicated to the object type “Port” (optical transceiver). Figure 3: Blocks responsible for mapping. Figure 2: FlowFile Processors responsible for loading the data into Apache NiFi. tributes_ Processor was responsible for extracting values of the defined JSON fields and saving them as the attributes of a FlowFile. The attribute name would be taken from the _Property_ column.