Project: Antrea (Runtime / Cloud Native Network)
File: network-flow-visibility.md

<!-- toc --> - - - - - - - - - - - - - - - - <!-- /toc --> is a Kubernetes network plugin that provides network connectivity and security features for Pod workloads. Considering the scale and dynamism of Kubernetes workloads in a cluster, Network Flow Visibility helps in the management and configuration of Kubernetes resources such as Network Policy, Services, Pods etc., and thereby provides opportunities to enhance the performance and security aspects of Pod workloads. For visualizing the network flows, Antrea monitors the flows in Linux conntrack module. These flows are converted to flow records, and then flow records are post-processed before they are sent to the configured external flow collector. High-level design is given below: In Antrea, the basic building block for the Network Flow Visibility is the Flow Exporter. Flow Exporter operates within Antrea Agent; it builds and maintains a connection store by polling and dumping flows from conntrack module periodically. Connections from the connection store are exported to the [Flow Aggregator Service](#flow-aggregator) using the IPFIX protocol, and for this purpose we use the IPFIX exporter process from the library. In addition to enabling the Flow Exporter feature gate (if needed), you need to ensure that the `flowExporter.enable` flag is set to true in the Antrea Agent configuration. your `antrea-agent` ConfigMap should look like this: ```yaml antrea-agent.conf: | featureGates: FlowExporter: true flowExporter: enable: true flowCollectorAddr: "flow-aggregator/flow-aggregator:4739:tls" flowPollInterval: "5s" activeFlowExportTimeout: "5s" idleFlowExportTimeout: "15s" ``` Please note that the default value for `flowExporter.flowCollectorAddr` is `"flow-aggregator/flow-aggregator:4739:tls"`, which enables the Flow Exporter to connect the Flow Aggregator Service, assuming it is running in the same K8 cluster with the Name and Namespace set to `flow-aggregator`. If you deploy the Flow Aggregator Service with a different Name and Namespace, then set `flowExporter.flowCollectorAddr` appropriately. Please note that the default values for `flowExporter.flowPollInterval`, `flowExporter.activeFlowExportTimeout`, and `flowExporter.idleFlowExportTimeout` parameters are set to 5s, 5s, and 15s, respectively. TLS communication between the Flow Exporter and the Flow Aggregator is enabled by default. Please modify them as per your requirements. Prior to the Antrea v1.13 release, the `flowExporter` option group in the Antrea Agent configuration did not exist. To enable the Flow Exporter feature, one simply needed to enable the feature gate, and the Flow Exporter related configuration could be configured using the (now deprecated) `flowCollectorAddr`, `flowPollInterval`, `activeFlowExportTimeout`, `idleFlowExportTimeout` parameters. There are 34 IPFIX IEs in each exported flow record, which are defined in the IANA-assigned IE registry, the Reverse IANA-assigned IE registry and the Antrea IE registry. The reverse IEs are used to provide bi-directional information about the flow. The Enterprise ID is 0 for IANA-assigned IE registry, 29305 for reverse IANA IE registry, 56505 for Antrea IE

All the IEs used by the Antrea Flow Exporter are listed below: | IPFIX Information Element| Field ID | Type | |--|-|-| | flowStartSeconds | 150 | dateTimeSeconds| | flowEndSeconds | 151 | dateTimeSeconds| | flowEndReason | 136 | unsigned8 | | sourceIPv4Address | 8 | ipv4Address | | destinationIPv4Address | 12 | ipv4Address | | sourceIPv6Address | 27 | ipv6Address | | destinationIPv6Address | 28 | ipv6Address | | sourceTransportPort | 7 | unsigned16 | | destinationTransportPort | 11 | unsigned16 | | protocolIdentifier | 4 | unsigned8 | | packetTotalCount | 86 | unsigned64 | | octetTotalCount | 85 | unsigned64 | | packetDeltaCount | 2 | unsigned64 | | octetDeltaCount | 1 | unsigned64 | | IPFIX Information Element| Field ID | Type | |--|-|-| | reversePacketTotalCount | 86 | unsigned64 | | reverseOctetTotalCount | 85 | unsigned64 | | reversePacketDeltaCount | 2 | unsigned64 | | reverseOctetDeltaCount | 1 | unsigned64 | | IPFIX Information Element | Field ID | Type | Description | |-|-|-|-| | sourcePodNamespace | 100 | string | | | sourcePodName | 101 | string | | | destinationPodNamespace | 102 | string | | | destinationPodName | 103 | string | | | sourceNodeName | 104 | string | | | destinationNodeName | 105 | string | | | destinationClusterIPv4 | 106 | ipv4Address | | | destinationClusterIPv6 | 107 | ipv6Address | | | destinationServicePort | 108 | unsigned16 | | | destinationServicePortName | 109 | string | | | ingressNetworkPolicyName | 110 | string | Name of the ingress network policy applied to the destination Pod for this flow. | | ingressNetworkPolicyNamespace | 111 | string | Namespace of the ingress network policy applied to the destination Pod for this flow. | | ingressNetworkPolicyType | 115 | unsigned8 | 1 stands for Kubernetes Network Policy. 2 stands for Antrea Network Policy. 3 stands for Antrea Cluster Network Policy. | | ingressNetworkPolicyRuleName | 141 | string | Name of the ingress network policy Rule applied to the destination Pod for this flow. | | egressNetworkPolicyName | 112 | string | Name of the egress network policy applied to the source Pod for this flow. | | egressNetworkPolicyNamespace | 113 | string | Namespace of the egress network policy applied to the source Pod for this flow. | | egressNetworkPolicyType | 118 | unsigned8 | | | egressNetworkPolicyRuleName | 142 | string | Name of the egress network policy rule applied to the source Pod for this flow. | | ingressNetworkPolicyRuleAction | 139 | unsigned8 | 1 stands for Allow. 2 stands for Drop. 3 stands for Reject. | | egressNetworkPolicyRuleAction | 140 | unsigned8 | | | tcpState | 136 | string | The state of the TCP connection. The states are: LISTEN, SYN-SENT, SYN-RECEIVED, ESTABLISHED, FIN-WAIT-1, FIN-WAIT-2, CLOSE-WAIT, CLOSING, LAST-ACK, TIME-WAIT, and CLOSED. | | flowType | 137 | unsigned8 | 1 stands for Intra-Node. 2 stands for Inter-Node. 3 stands for To External. 4 stands for From External. | Currently, the Flow Exporter feature provides visibility for Pod-to-Pod, Pod-to-Service and Pod-to-External network flows along with the associated statistics such as data throughput (bits per second), packet throughput (packets per second), cumulative byte count and cumulative packet count. Pod-To-Service flow visibility is supported only , which is the case by default starting with Antrea v0.11. In the future, we will enable the support for External-To-Service flows. Kubernetes information such as Node name, Pod name, Pod Namespace, Service name, NetworkPolicy name and NetworkPolicy Namespace, is added to the flow records. Network Policy Rule Action (Allow, Reject, Drop) is also supported for both Antrea-native NetworkPolicies and K8s NetworkPolicies. For K8s NetworkPolicies, connections dropped due to will be assigned the Drop action. For flow records that are exported from any given Antrea Agent, the Flow Exporter only provides the information of Kubernetes entities that are local to the Antrea Agent. In other words, flow records are only complete for intra-Node flows, but incomplete for inter-Node flows. It is the responsibility of the to correlate flows from the source and destination Nodes and produce complete flow records. Both Flow Exporter and Flow Aggregator are supported in IPv4 clusters, IPv6 clusters and dual-stack clusters. We support following connection metrics as Prometheus metrics that are exposed through : `antreaagentconntracktotalconnection_count`, `antreaagentconntrackantreaconnection_count`, `antreaagentdeniedconnectioncount`, `antreaagentconntrackmaxconnection_count`, and `antreaagentflowcollectorreconnection_count` Flow Aggregator is deployed as a Kubernetes Service. The main functionality of Flow Aggregator is to store, correlate and aggregate the flow records received from the Flow Exporter of Antrea

More details on the functionality are provided in the section. Flow Aggregator is implemented as IPFIX mediator, which consists of IPFIX Collector Process, IPFIX Intermediate Process and IPFIX Exporter Process. We use the library to implement the Flow Aggregator. To deploy a released version of Flow Aggregator Service, pick a deployment manifest from the . For any given release `<TAG>` (e.g. `v0.12.0`), you can deploy Flow Aggregator as follows: ```bash kubectl apply -f https://github.com/antrea-io/antrea/releases/download/<TAG>/flow-aggregator.yml ``` To deploy the latest version of Flow Aggregator Service (built from the main branch), use the checked-in : ```bash kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/main/build/yamls/flow-aggregator.yml ``` The following configuration parameters have to be provided through the Flow Aggregator ConfigMap. Flow aggregator needs to be configured with at least one of the supported . `flowCollector` is mandatory for , and `clickHouse` is mandatory for . We provide an example value for this parameter in the following snippet. If you have deployed the , then please set `flowCollector.enable` to `true` and use the address for `flowCollector.address`: `<Ipfix-Collector Cluster IP>:<port>:<tcp|udp>` If you have deployed the , then please enable the collector by setting `clickHouse.enable` to `true`. If it is deployed following the , the ClickHouse server is already exposed via a K8s Service, and no further configuration is required. If a different FQDN or IP is desired, please use the URL for `clickHouse.databaseURL` in the following format: `<protocol>://<ClickHouse server FQDN or IP>:<ClickHouse port>`. Starting with Antrea v1.13, you can enable TLS when connecting to the ClickHouse Server by setting `clickHouse.databaseURL` with protocol `tls` or `https`. You can also change the value of `clickHouse.tls.insecureSkipVerify` to determine whether to skip the verification of the server's certificate. If you want to provide a custom CA certificate, you can set `clickHouse.tls.caCert` to `true` and the flow Aggregator will read the certificate key pair from the`clickhouse-ca` Secret. Make sure to follow the following form when creating the `clickhouse-ca` Secret with the custom CA certificate: ```yaml apiVersion: v1 kind: Secret metadata: name: clickhouse-ca namespace: flow-aggregator data: ca.crt: <BASE64 ENCODED CA CERTIFICATE> ``` You can use `kubectl apply -f <PATH TO SECRET YAML>` to create the above secret , or use `kubectl create secret`: ```bash kubectl create secret generic clickhouse-ca -n flow-aggregator --from-file=ca.crt=<PATH TO CA CERTIFICATE> ``` Prior to Antrea v1.13, secure connections to ClickHouse are not supported, and TCP is the only supported protocol when connecting to the ClickHouse server from the Flow Aggregator. ```yaml flow-aggregator.conf: | activeFlowRecordTimeout: 60s inactiveFlowRecordTimeout: 90s aggregatorTransportProtocol: "tls" flowAggregatorAddress: "" recordContents: podLabels: false apiServer: apiPort: 10348 tlsCipherSuites: "" tlsMinVersion: "" flowCollector: enable: false address: "" recordFormat: "IPFIX" clickHouse: enable: false database: "default" databaseURL: "tcp://clickhouse-clickhouse.flow-visibility.svc:9000" tls: insecureSkipVerify: false caCert: false debug: false compress: true commitInterval: "8s" ``` Please note that the default values for `activeFlowRecordTimeout`, `inactiveFlowRecordTimeout`, `aggregatorTransportProtocol` parameters are set to `60s`, `90s` and `tls` respectively. Please make sure that `aggregatorTransportProtocol` and protocol of `flowCollectorAddr` in `agent-agent.conf` are set to `tls` to guarantee secure communication works properly. Protocol of `flowCollectorAddr` and `aggregatorTransportProtocol` must always match, so TLS must either be enabled for both sides or disabled for both sides. Please modify the parameters as per your requirements. Please note that the default value for `recordContents.podLabels` is `false`, which indicates source and destination Pod labels will not be included in the flow records exported to `flowCollector` and `clickHouse`. If you would like to include them, you can modify the value to `true`. Please note that the default value for `apiServer.apiPort` is `10348`, which is the port used to expose the Flow Aggregator's APIServer. Please modify the parameters as per your requirements. Please note that the default value for

is `8s`, which is based on experiment results to achieve best ClickHouse write performance and data retention. Based on ClickHouse recommendation for best performance, this interval is required be no shorter than `1s`. Also note that flow aggregator has a cache limit of ~500k records for ClickHouse-Grafana collector. If `clickHouse.commitInterval` is set to a value too large, there's a risk of losing records. In addition to IPFIX information elements provided in the , the Flow Aggregator adds the following fields to the flow records. | IPFIX Information Element | Field ID | Type | Description | |-|-|-|-| | packetTotalCountFromSourceNode | 120 | unsigned64 | The cumulative number of packets for this flow as reported by the source Node, since the flow started. | | octetTotalCountFromSourceNode | 121 | unsigned64 | The cumulative number of octets for this flow as reported by the source Node, since the flow started. | | packetDeltaCountFromSourceNode | 122 | unsigned64 | The number of packets for this flow as reported by the source Node, since the previous report for this flow at the observation point. | | octetDeltaCountFromSourceNode | 123 | unsigned64 | The number of octets for this flow as reported by the source Node, since the previous report for this flow at the observation point. | | reversePacketTotalCountFromSourceNode | 124 | unsigned64 | The cumulative number of reverse packets for this flow as reported by the source Node, since the flow started. | | reverseOctetTotalCountFromSourceNode | 125 | unsigned64 | The cumulative number of reverse octets for this flow as reported by the source Node, since the flow started. | | reversePacketDeltaCountFromSourceNode | 126 | unsigned64 | The number of reverse packets for this flow as reported by the source Node, since the previous report for this flow at the observation point. | | reverseOctetDeltaCountFromSourceNode | 127 | unsigned64 | The number of reverse octets for this flow as reported by the source Node, since the previous report for this flow at the observation point. | | packetTotalCountFromDestinationNode | 128 | unsigned64 | The cumulative number of packets for this flow as reported by the destination Node, since the flow started. | | octetTotalCountFromDestinationNode | 129 | unsigned64 | The cumulative number of octets for this flow as reported by the destination Node, since the flow started. | | packetDeltaCountFromDestinationNode | 130 | unsigned64 | The number of packets for this flow as reported by the destination Node, since the previous report for this flow at the observation point. | | octetDeltaCountFromDestinationNode | 131 | unsigned64 | The number of octets for this flow as reported by the destination Node, since the previous report for this flow at the observation point. | | reversePacketTotalCountFromDestinationNode| 132 | unsigned64 | The cumulative number of reverse packets for this flow as reported by the destination Node, since the flow started. | | reverseOctetTotalCountFromDestinationNode | 133 | unsigned64 | The cumulative number of reverse octets for this flow as reported by the destination Node, since the flow started. | | reversePacketDeltaCountFromDestinationNode| 134 | unsigned64 | The number of reverse packets for this flow as reported by the destination Node, since the previous report for this flow at the observation point. | | reverseOctetDeltaCountFromDestinationNode | 135 | unsigned64 | The number of reverse octets for this flow as reported by the destination Node, since the previous report for this flow at the observation

| | sourcePodLabels | 143 | string | | | destinationPodLabels | 144 | string | | | throughput | 145 | unsigned64 | The average amount of traffic flowing from source to destination, since the previous report for this flow at the observation point. The unit is bits per second. | | reverseThroughput | 146 | unsigned64 | The average amount of reverse traffic flowing from destination to source, since the previous report for this flow at the observation point. The unit is bits per second. | | throughputFromSourceNode | 147 | unsigned64 | The average amount of traffic flowing from source to destination, since the previous report for this flow at the observation point, based on the records sent from the source Node. The unit is bits per second. | | throughputFromDestinationNode | 148 | unsigned64 | The average amount of traffic flowing from source to destination, since the previous report for this flow at the observation point, based on the records sent from the destination Node. The unit is bits per second. | | reverseThroughputFromSourceNode | 149 | unsigned64 | The average amount of reverse traffic flowing from destination to source, since the previous report for this flow at the observation point, based on the records sent from the source Node. The unit is bits per second. | | reverseThroughputFromDestinationNode | 150 | unsigned64 | The average amount of reverse traffic flowing from destination to source, since the previous report for this flow at the observation point, based on the records sent from the destination Node. The unit is bits per second. | | flowEndSecondsFromSourceNode | 151 | unsigned32 | The absolute timestamp of the last packet of this flow, based on the records sent from the source Node. The unit is seconds. | | flowEndSecondsFromDestinationNode | 152 | unsigned32 | The absolute timestamp of the last packet of this flow, based on the records sent from the destination Node. The unit is seconds. | Flow Aggregator stores the received flow records from Antrea Agents in a hash map, where the flow key is 5-tuple of a network connection. 5-tuple consists of Source IP, Destination IP, Source Port, Destination Port and Transport protocol. Therefore, Flow Aggregator maintains one flow record for any given connection, and this flow record gets updated till the connection in the Kubernetes cluster becomes invalid. In the case of inter-Node flows, there are two flow records, one from the source Node, where the flow originates from, and another one from the destination Node, where the destination Pod resides. Both the flow records contain incomplete information as mentioned . Flow Aggregator provides support for the correlation of the flow records from the source Node and the destination Node, and it exports a single flow record with complete information for both inter-Node and intra-Node flows. Flow Aggregator aggregates the flow records that belong to a single connection. As part of aggregation, fields such as flow timestamps, flow statistics etc. are updated. For the purpose of updating flow statistics fields, Flow Aggregator introduces the in Antrea Enterprise IPFIX registry corresponding to the Source Node and Destination Node, so that flow statistics from different Nodes can be preserved. antctl can access the Flow Aggregator API to dump flow records and print metrics about flow record processing. Refer to the for more information. If you would like to quickly try Network Flow Visibility feature, you can deploy Antrea, the Flow Aggregator Service, the Grafana Flow Collector on the . Build required image under antrea by using make command: ```shell make make flow-aggregator-image ``` Given any external IPFIX flow collector, you can deploy Antrea and the Flow Aggregator Service on a default Vagrant setup by running the following commands: ```shell ./infra/vagrant/provision.sh

--flow-collector <externalFlowCollectorAddress> ``` If you would like to deploy the Grafana Flow Collector, you can run the following command: ```shell ./infra/vagrant/provision.sh ./infra/vagrant/push_antrea.sh --flow-collector Grafana ``` Here we list two choices the external configured flow collector: go-ipfix collector and Grafana flow collector. For each collector, we introduce how to deploy it and how to output or visualize the collected flow records information. The go-ipfix collector can be built from . It is used to collect, decode and log the IPFIX records. To deploy a released version of the go-ipfix collector, please choose one deployment manifest from the list of releases (supported after v0.5.2). For any given release <TAG> (e.g. v0.5.2), you can deploy the collector as follows: ```shell kubectl apply -f https://github.com/vmware/go-ipfix/releases/download/<TAG>/ipfix-collector.yaml ``` To deploy the latest version of the go-ipfix collector (built from the main branch), use the checked-in : ```shell kubectl apply -f https://raw.githubusercontent.com/vmware/go-ipfix/main/build/yamls/ipfix-collector.yaml ``` Go-ipfix collector also supports customization on its parameters: port and protocol. Please follow the to configure those parameters if needed. To output the flow records collected by the go-ipfix collector, use the command below: ```shell kubectl logs <ipfix-collector-pod-name> -n ipfix ``` Starting with Antrea v1.8, support for the Grafana Flow Collector has been migrated to Theia. The Grafana Flow Collector was added in Antrea v1.6.0. In Antrea v1.7.0, we start to move the network observability and analytics functionalities of Antrea to , including the Grafana Flow Collector. Going forward, further development of the Grafana Flow Collector will be in the Theia repo. For the up-to-date version of Grafana Flow Collector and other Theia features, please refer to the . Starting with Antrea v1.7, support for the ELK Flow Collector has been removed. Please consider using the instead, which is actively maintained. In addition to layer 4 network visibility, Antrea adds layer 7 network flow export. To achieve L7 (Layer 7) network flow export, the `L7FlowExporter` feature gate must be enabled. Note: L7 flow-visibility support for Theia is not yet implemented. To export layer 7 flows of a Pod or a Namespace, user can annotate Pods or Namespaces with the annotation key `visibility.antrea.io/l7-export` and set the value to indicate the traffic flow direction, which can be `ingress`, `egress` or `both`. For example, to enable L7 flow export in the ingress direction on Pod test-pod in the default Namespace, you can use: ```bash kubectl annotate pod test-pod visibility.antrea.io/l7-export=ingress ``` Based on the annotation, Flow Exporter will export the L7 flow data to the Flow Aggregator or configured IPFix collector using the fields `appProtocolName` and `httpVals`. `appProtocolName` field is used to indicate the application layer protocol name (e.g. http) and it will be empty if application layer data is not exported. `httpVals` stores a serialized JSON dictionary with every HTTP request for a connection mapped to a unique transaction ID. This format lets us group all the HTTP transactions pertaining to the same connection, into the same exported record. An example of `httpVals` is : `"{\"0\":{\"hostname\":\"10.10.0.1\",\"url\":\"/public/\",\"httpuseragent\":\"curl/7.74.0\",\"httpcontenttype\":\"text/html\",\"http_method\":\"GET\",\"protocol\":\"HTTP/1.1\",\"status\":200,\"length\":153}}"` HTTP fields in the `httpVals` are: | Http field | Description | |-|--| | hostname | IP address of the sender | | URL | url requested on the server | | httpuseragent | application used for HTTP | | httpcontenttype | type of content being returned by the server | | http_method | HTTP method used for the request | | protocol | HTTP protocol version used for the request or response | | status | HTTP status code | | length | size of the response body | As of now, the only supported layer 7 protocol is `HTTP1.1`. Support for more protocols may be added in the future.