# SmartX Multi-View Visibility Framework
## 3 SmartX MVF with Flow-Centric Visibility: Design and Implementation
### Visibility Data Aggregation and Tagging  
To satisfy visibility data aggregation and tagging requirements, flow-centric visibility has a special component. Under the title of SmartX tagging, additional information is collected from playground users and system generated information is extracted from pro and ID centers. Figure 5 shows the various tags that are acquired and incorporated into visibility data. Visibility data aggregation requirement is also facilitated by this component. Visibility data is aggregated based on the statistical values at five-minute intervals by using average function against the key identifiers such as SmartX box ID. In addition, visibility data aggregation also helps in overcoming the visibility data processing dilemma of distributed playground resources locality and data timestamp issues. For systematic processing of visibility data, SmartX MVF employs the Apache Spark analytics engine [39]. Core advantages of Spark are its speed of operation, native support for selected datastores of DataLake, and extensive API support.  
The design of visibility data aggregation and tagging component is shown in Figure 6. There are four main functions that facilitate flexible processing of multi-layer visibility data. An aggregate  
Figure 4: Design of network packets flow collection component.  
Figure 5: Automatically acquired SmartX tags from the playground tower and playground users.  
underlay data function loads visibility data from the MongoDB datastore [13] of DataLake. Query and filter specify conditions for matching data and pushes the queries to the MongoDB to bring only matching data in the Spark executor program. On the fetched data, grouping function average is applied to reduce the volume of data. Before saving the resultant data frame to the Elasticsearch [13] and Parquet [40] datastores, a timestamp field is added. A dataset is a distributed collection of data and a data frame is a dataset organized into named columns. Elasticsearch datastore retains processed visibility data for long-term needs and Parquet datastore keeps the lastly processed visibility data and is consumed by a subsequent component of visibility integration stage. Besides raw data filtering and aggregation steps, aggregate physical data function also incorporates SmartX tags to the system level visibility metrics. Aggregate packets data function deals with flow-layer visibility data from each SmartX box. A SmartX box has two data files. The first file contains management and control planes network packets and the second file contains data plane network packets. Aggregate packet data function gets all the files from the disk and creates a combined data frame. SmartX tags are then incorporated into this data according to the matching identifiers in the data frame. Additional statistical fields are extracted (e.g., min/max/avg/total data bytes) for more detailed information about playground network traffic.  
Apache Spark version 2.2.0 and Java version 1.8.0_201 are installed on vis center. SmartX tags acquisition is implemented by shell scripting and OpenStack cloud platform python APIs and execute at various centers of playground tower. Visibility data tagging and aggregation component is developed using the Scala programming language. By using simple build tool (sbt) [41], a thin jar file is created and deployed at the vis center. To repeatedly execute this component, we developed a shell script and following command specifies execution parameters when running this component.  
SSPARK_HOME/bin/spark-submit -class smartx.multiview.flowcentric.Main -master local[*] -driver-memory 24g multi-view-flowcentric-tag-aggregate_2.11-0.1.jar \"x.x.x.x\" &&  
This command utilizes spark-submit tool to execute data processing jobs on the local system with all available CPU cores and 24G of memory. Other important parameters are the class that specifies the name of the main Scala class and the name of the executable jar file.