## 3 Proposed Solution
### Flow-Centric Visibility to Assist Networking Control  
Flow-centric visibility plays a crucial role in verifying packets flowing through the distributed sites and virtualized overlay networks, and checking the payload associated with specific workloads, such as tenant applications. While there are various flow-based visibility solutions available with distinct approaches for collecting and processing network packets, the majority of these are commercial solutions that require exclusive hardware and licensed software. Thus, we design a customized flow-centric visibility solution according to the distributed multi-site cloud monitor and control requirements. Our design employs a combination of methods and open-source tools, building upon the SmartX MVF [3], and consists of four key stages:  
* _Visibility collection, messaging, and validation_ involves collecting visibility metrics from the distributed Linux servers and validating them before forwarding to the visibility center. The collected data include packet attributes such as IP addresses, packet sizes, and types, which are essential for subsequent analysis. * _Visibility integration_ performs flow clustering and identification based on collected packet data. This stage involves grouping flows into multiple classes using flow attributes and behaviors, and mapping them to predefined policies. The classes include: 1. Clustered flows: flows with similar packet attributes, such as source/destination subnets, packet size, and TCP header type. 2. Identified flows: flows generated by specific applications and tenants, identified using operator-provided tags. 3. Known flows: flows that belong to a known tenant but lack application info. 4. Unknown flows: flows with unknown IP addresses and random port numbers, which remain unclassified after analysis. 5. Attack flows: flows from sources known to be responsible for attacking other hosts and engaging in malicious behavior. Operator-provided tags, offered by cloud operators and automation tools, are fundamental in this stage to assist in flow clustering and identification, especially since SmartX MVF does not support application-level visibility natively. * _Visibility DataLake_ stores numerous types and volumes of raw, integrated, and staged visibility data in different formats. This DataLake acts as a central repository, supporting convenient data retrieval and analysis. * _Visibility visualization_ facilitates access to the analyzed data and transforms them into graphical outputs for easy interpretation. This stage includes dashboards and reports that provide insights into network performance and security status. The visibility integration stage is pivotal because it generates actionable insights by classifying flows into the aforementioned categories. For instance, flows are clustered based on the source/destination subnets, determined by analyzing IP addresses and subnet masks. These clusters undergo statistical analysis to identify suspicious patterns, which may indicate security issues. Identified flows are associated with known applications and tenants, leveraging tags for accuracy. Unknown flows, not matching any known application patterns, are categorized separately for further investigation. Periodic analysis of flows can mark some flows as suspicious due to behaviors or statistics of the bytes number, packet type, and sending patterns, which may lead to a security incident. However, it is important to remember that, even after flow-centric visibility analysis, some flows may remain as unknown flows (i.e., unknown IP address and random port number). This multi-class flow information approach effectively reduces the volume of visibility data that need to be analyzed for networking control. For instance, we cluster flows based on source/destination subnets, which are determined based on the IP source/destination, with additional subnet mask information provided by tagging.