## 4 Implementation
### Flow-Centric Visibility to Assist Networking Control  
Our flow-centric visibility solution is self-developed by leveraging various open-source software and programming languages. By leveraging these technologies and programming languages, we have developed a robust flow-centric visibility solution. Each component plays a vital role in this solution, contributing to the efficient collection, integration, storage, and visualization of network packet flow data. Here is an overview of the implementation details for each component:  
* The visibility collection and validation solution is developed using the Java programming language. It publishes traced data from the IO Visor-based packet tracing solution to the Kafka messaging system [22]. This solution also validates the data format and values before transferring the data in Kafka queues for further processing. * Visibility integration is implemented using the Scala programming language with Apache Spark libraries [23] to (1) aggregate the similar or duplicated data, (2) tag the data with predefined configuration (i.e., source devices and site name), (3) cluster the data into summarized attributes (i.e., subnet IP), (4) identify the data for each tenant and application name/type, and (5) generate the recommended policies for assisting the networking control. Listing 2 shows a command from our bash script highlighting how the flow classification application is deployed over a standalone Spark cluster. This command efficiently submits a Spark job to classify network flow data using the _smartx_._mrf_classify_Main_ class from the specified JAR file. It runs locally with configurations for driver and executor resources, allocating a specified number of cores and memory to each. This precise resource allocation is to control and ensure optimal performance. The command also enables Spark event logging to a local directory, providing a comprehensive view of the system's operations. It processes the input network packet flow records and attacker information, and stores the classified results in the designated output location, further reinforcing the system's efficiency. * Visibility DataLake is leveraging the Apache Parquet [24] data file format to store and retrieve processed data in an efficient manner. For the staged data, an index-based DataS-tore, named Elasticsearch [25], is used to store the results of clustering and identification. * Visibility visualization leverages the web-based user interface, built up with the JavaScript programming language. The development of the visibility integration is one of the critical aspects of our solution, as it deals with diverse types of data and configuration inputs. Listing 3 provides an overview of the visibility integration process, illustrating how packets are aggregated and then clustered into classes such as identified, unknown, or suspicious. ```
spark-submit-classsmartx.mvf.classify.Main \ --masterspark://master.7077-deploy-modeluser \ -driver-cores2-.driver-memory4g \ --executor-cores2-executor-memory2g \ --total-executor-cores2-num-executors1 \ mvf-flow-classification-assembly-0.1.0.jar \ /path/to/flow_recordsflows-classifiedattackers.csv
```  
Listing 2: Submitting flow-centric application to Spark cluster.