Chapter 7. Observing the network traffic As an administrator, you can observe the network traffic in the OpenShift Container Platform console for detailed troubleshooting and analysis. This feature helps you get insights from different graphical representations of traffic flow. There are several available views to observe the network traffic. 7.1. Observing the network traffic from the Overview view The Overview view displays the overall aggregated metrics of the network traffic flow on the cluster. As an administrator, you can monitor the statistics with the available display options. 7.1.1. Working with the Overview view As an administrator, you can navigate to the Overview view to see the graphical representation of the flow rate statistics. Procedure Navigate to Observe Network Traffic . In the Network Traffic page, click the Overview tab. You can configure the scope of each flow rate data by clicking the menu icon. 7.1.2. Configuring advanced options for the Overview view You can customize the graphical view by using advanced options. To access the advanced options, click Show advanced options . You can configure the details in the graph by using the Display options drop-down menu. The options available are as follows: Scope : Select to view the components that network traffic flows between. You can set the scope to Node , Namespace , Owner , Zones , Cluster or Resource . Owner is an aggregation of resources. Resource can be a pod, service, node, in case of host-network traffic, or an unknown IP address. The default value is Namespace . Truncate labels : Select the required width of the label from the drop-down list. The default value is M . 7.1.2.1. Managing panels and display You can select the required panels to be displayed, reorder them, and focus on a specific panel. To add or remove panels, click Manage panels . The following panels are shown by default: Top X average bytes rates Top X bytes rates stacked with total Other panels can be added in Manage panels : Top X average packets rates Top X packets rates stacked with total Query options allows you to choose whether to show the Top 5 , Top 10 , or Top 15 rates. 7.1.2.2. DNS tracking You can configure graphical representation of Domain Name System (DNS) tracking of network flows in the Overview view. Using DNS tracking with extended Berkeley Packet Filter (eBPF) tracepoint hooks can serve various purposes: Network Monitoring: Gain insights into DNS queries and responses, helping network administrators identify unusual patterns, potential bottlenecks, or performance issues. Security Analysis: Detect suspicious DNS activities, such as domain name generation algorithms (DGA) used by malware, or identify unauthorized DNS resolutions that might indicate a security breach. Troubleshooting: Debug DNS-related issues by tracing DNS resolution steps, tracking latency, and identifying misconfigurations. By default, when DNS tracking is enabled, you can see the following non-empty metrics represented in a donut or line chart in the Overview : Top X DNS Response Code Top X average DNS latencies with overall Top X 90th percentile DNS latencies Other DNS tracking panels can be added in Manage panels : Bottom X minimum DNS latencies Top X maximum DNS latencies Top X 99th percentile DNS latencies This feature is supported for IPv4 and IPv6 UDP and TCP protocols. See the Additional resources in this section for more information about enabling and working with this view. Additional resources Working with DNS tracking Network Observability metrics 7.1.3. Round-Trip Time You can use TCP smoothed Round-Trip Time (sRTT) to analyze network flow latencies. You can use RTT captured from the fentry/tcp_rcv_established eBPF hookpoint to read sRTT from the TCP socket to help with the following: Network Monitoring: Gain insights into TCP latencies, helping network administrators identify unusual patterns, potential bottlenecks, or performance issues. Troubleshooting: Debug TCP-related issues by tracking latency and identifying misconfigurations. By default, when RTT is enabled, you can see the following TCP RTT metrics represented in the Overview : Top X 90th percentile TCP Round Trip Time with overall Top X average TCP Round Trip Time with overall Bottom X minimum TCP Round Trip Time with overall Other RTT panels can be added in Manage panels : Top X maximum TCP Round Trip Time with overall Top X 99th percentile TCP Round Trip Time with overall See the Additional resources in this section for more information about enabling and working with this view. Additional resources Working with RTT tracing 7.1.4. eBPF flow rule filter You can use rule-based filtering to control the volume of packets cached in the eBPF flow table. For example, a filter can specify that only packets coming from port 100 should be recorded. Then only the packets that match the filter are cached and the rest are not cached. 7.1.4.1. Ingress and egress traffic filtering CIDR notation efficiently represents IP address ranges by combining the base IP address with a prefix length. For both ingress and egress traffic, the source IP address is first used to match filter rules configured with CIDR notation. If there is a match, then the filtering proceeds. If there is no match, then the destination IP is used to match filter rules configured with CIDR notation. After matching either the source IP or the destination IP CIDR, you can pinpoint specific endpoints using the peerIP to differentiate the destination IP address of the packet. Based on the provisioned action, the flow data is either cached in the eBPF flow table or not cached. 7.1.4.2. Dashboard and metrics integrations When this option is enabled, the Netobserv/Health dashboard for eBPF agent statistics now has the Filtered flows rate view. Additionally, in Observe Metrics you can query netobserv_agent_filtered_flows_total to observe metrics with the reason in FlowFilterAcceptCounter , FlowFilterNoMatchCounter or FlowFilterRecjectCounter . 7.1.4.3. Flow filter configuration parameters The flow filter rules consist of required and optional parameters. Table 7.1. Required configuration parameters Parameter Description enable Set enable to true to enable the eBPF flow filtering feature. cidr Provides the IP address and CIDR mask for the flow filter rule. Supports both IPv4 and IPv6 address format. If you want to match against any IP, you can use 0.0.0.0/0 for IPv4 or ::/0 for IPv6. action Describes the action that is taken for the flow filter rule. The possible values are Accept or Reject . For the Accept action matching rule, the flow data is cached in the eBPF table and updated with the global metric, FlowFilterAcceptCounter . For the Reject action matching rule, the flow data is dropped and not cached in the eBPF table. The flow data is updated with the global metric, FlowFilterRejectCounter . If the rule is not matched, the flow is cached in the eBPF table and updated with the global metric, FlowFilterNoMatchCounter . Table 7.2. Optional configuration parameters Parameter Description direction Defines the direction of the flow filter rule. Possible values are Ingress or Egress . protocol Defines the protocol of the flow filter rule. Possible values are TCP , UDP , SCTP , ICMP , and ICMPv6 . tcpFlags Defines the TCP flags to filter flows. Possible values are SYN , SYN-ACK , ACK , FIN , RST , PSH , URG , ECE , CWR , FIN-ACK , and RST-ACK . ports Defines the ports to use for filtering flows. It can be used for either source or destination ports. To filter a single port, set a single port as an integer value. For example ports: 80 . To filter a range of ports, use a "start-end" range in string format. For example ports: "80-100" sourcePorts Defines the source port to use for filtering flows. To filter a single port, set a single port as an integer value, for example sourcePorts: 80 . To filter a range of ports, use a "start-end" range, string format, for example sourcePorts: "80-100" . destPorts DestPorts defines the destination ports to use for filtering flows. To filter a single port, set a single port as an integer value, for example destPorts: 80 . To filter a range of ports, use a "start-end" range in string format, for example destPorts: "80-100" . icmpType Defines the ICMP type to use for filtering flows. icmpCode Defines the ICMP code to use for filtering flows. peerIP Defines the IP address to use for filtering flows, for example: 10.10.10.10 . Additional resources Filtering eBPF flow data with rules Network Observability metrics Health dashboards 7.2. Observing the network traffic from the Traffic flows view The Traffic flows view displays the data of the network flows and the amount of traffic in a table. As an administrator, you can monitor the amount of traffic across the application by using the traffic flow table. 7.2.1. Working with the Traffic flows view As an administrator, you can navigate to Traffic flows table to see network flow information. Procedure Navigate to Observe Network Traffic . In the Network Traffic page, click the Traffic flows tab. You can click on each row to get the corresponding flow information. 7.2.2. Configuring advanced options for the Traffic flows view You can customize and export the view by using Show advanced options . You can set the row size by using the Display options drop-down menu. The default value is Normal . 7.2.2.1. Managing columns You can select the required columns to be displayed, and reorder them. To manage columns, click Manage columns . 7.2.2.2. Exporting the traffic flow data You can export data from the Traffic flows view. Procedure Click Export data . In the pop-up window, you can select the Export all data checkbox to export all the data, and clear the checkbox to select the required fields to be exported. Click Export . 7.2.3. Working with conversation tracking As an administrator, you can group network flows that are part of the same conversation. A conversation is defined as a grouping of peers that are identified by their IP addresses, ports, and protocols, resulting in an unique Conversation Id . You can query conversation events in the web console. These events are represented in the web console as follows: Conversation start : This event happens when a connection is starting or TCP flag intercepted Conversation tick : This event happens at each specified interval defined in the FlowCollector spec.processor.conversationHeartbeatInterval parameter while the connection is active. Conversation end : This event happens when the FlowCollector spec.processor.conversationEndTimeout parameter is reached or the TCP flag is intercepted. Flow : This is the network traffic flow that occurs within the specified interval. Procedure In the web console, navigate to Operators Installed Operators . Under the Provided APIs heading for the NetObserv Operator , select Flow Collector . Select cluster then select the YAML tab. Configure the FlowCollector custom resource so that spec.processor.logTypes , conversationEndTimeout , and conversationHeartbeatInterval parameters are set according to your observation needs. A sample configuration is as follows: Configure FlowCollector for conversation tracking apiVersion: flows.netobserv.io/v1beta2 kind: FlowCollector metadata: name: cluster spec: processor: logTypes: Flows 1 advanced: conversationEndTimeout: 10s 2 conversationHeartbeatInterval: 30s 3 1 When logTypes is set to Flows , only the Flow event is exported. If you set the value to All , both conversation and flow events are exported and visible in the Network Traffic page. To focus only on conversation events, you can specify Conversations which exports the Conversation start , Conversation tick and Conversation end events; or EndedConversations exports only the Conversation end events. Storage requirements are highest for All and lowest for EndedConversations . 2 The Conversation end event represents the point when the conversationEndTimeout is reached or the TCP flag is intercepted. 3 The Conversation tick event represents each specified interval defined in the FlowCollector conversationHeartbeatInterval parameter while the network connection is active. Note If you update the logType option, the flows from the selection do not clear from the console plugin. For example, if you initially set logType to Conversations for a span of time until 10 AM and then move to EndedConversations , the console plugin shows all conversation events before 10 AM and only ended conversations after 10 AM. Refresh the Network Traffic page on the Traffic flows tab. Notice there are two new columns, Event/Type and Conversation Id . All the Event/Type fields are Flow when Flow is the selected query option. Select Query Options and choose the Log Type , Conversation . Now the Event/Type shows all of the desired conversation events. you can filter on a specific conversation ID or switch between the Conversation and Flow log type options from the side panel. 7.2.4. Working with DNS tracking Using DNS tracking, you can monitor your network, conduct security analysis, and troubleshoot DNS issues. You can track DNS by editing the FlowCollector to the specifications in the following YAML example. Important CPU and memory usage increases are observed in the eBPF agent when this feature is enabled. Procedure In the web console, navigate to Operators Installed Operators . Under the Provided APIs heading for Network Observability , select Flow Collector . Select cluster then select the YAML tab. Configure the FlowCollector custom resource. A sample configuration is as follows: Configure FlowCollector for DNS tracking apiVersion: flows.netobserv.io/v1beta2 kind: FlowCollector metadata: name: cluster spec: namespace: netobserv agent: type: eBPF ebpf: features: - DNSTracking 1 sampling: 1 2 1 You can set the spec.agent.ebpf.features parameter list to enable DNS tracking of each network flow in the web console. 2 You can set sampling to a value of 1 for more accurate metrics and to capture DNS latency . For a sampling value greater than 1, you can observe flows with DNS Response Code and DNS Id , and it is unlikely that DNS Latency can be observed. When you refresh the Network Traffic page, there are new DNS representations you can choose to view in the Overview and Traffic Flow views and new filters you can apply. Select new DNS choices in Manage panels to display graphical visualizations and DNS metrics in the Overview . Select new choices in Manage columns to add DNS columns to the Traffic Flows view. Filter on specific DNS metrics, such as DNS Id , DNS Error DNS Latency and DNS Response Code , and see more information from the side panel. The DNS Latency and DNS Response Code columns are shown by default. Note TCP handshake packets do not have DNS headers. TCP protocol flows without DNS headers are shown in the traffic flow data with DNS Latency , ID , and Response code values of "n/a". You can filter out flow data to view only flows that have DNS headers using the Common filter "DNSError" equal to "0". 7.2.5. Working with RTT tracing You can track RTT by editing the FlowCollector to the specifications in the following YAML example. Procedure In the web console, navigate to Operators Installed Operators . In the Provided APIs heading for the NetObserv Operator , select Flow Collector . Select cluster , and then select the YAML tab. Configure the FlowCollector custom resource for RTT tracing, for example: Example FlowCollector configuration apiVersion: flows.netobserv.io/v1beta2 kind: FlowCollector metadata: name: cluster spec: namespace: netobserv agent: type: eBPF ebpf: features: - FlowRTT 1 1 You can start tracing RTT network flows by listing the FlowRTT parameter in the spec.agent.ebpf.features specification list. Verification When you refresh the Network Traffic page, the Overview , Traffic Flow , and Topology views display new information about RTT: In the Overview , select new choices in Manage panels to choose which graphical visualizations of RTT to display. In the Traffic flows table, the Flow RTT column can be seen, and you can manage display in Manage columns . In the Traffic Flows view, you can also expand the side panel to view more information about RTT. Example filtering Click the Common filters Protocol . Filter the network flow data based on TCP , Ingress direction, and look for FlowRTT values greater than 10,000,000 nanoseconds (10ms). Remove the Protocol filter. Filter for Flow RTT values greater than 0 in the Common filters. In the Topology view, click the Display option dropdown. Then click RTT in the edge labels drop-down list. 7.2.5.1. Using the histogram You can click Show histogram to display a toolbar view for visualizing the history of flows as a bar chart. The histogram shows the number of logs over time. You can select a part of the histogram to filter the network flow data in the table that follows the toolbar. 7.2.6. Working with availability zones You can configure the FlowCollector to collect information about the cluster availability zones. This allows you to enrich network flow data with the topology.kubernetes.io/zone label value applied to the nodes. Procedure In the web console, go to Operators Installed Operators . Under the Provided APIs heading for the NetObserv Operator , select Flow Collector . Select cluster then select the YAML tab. Configure the FlowCollector custom resource so that the spec.processor.addZone parameter is set to true . A sample configuration is as follows: Configure FlowCollector for availability zones collection apiVersion: flows.netobserv.io/v1beta2 kind: FlowCollector metadata: name: cluster spec: # ... processor: addZone: true # ... Verification When you refresh the Network Traffic page, the Overview , Traffic Flow , and Topology views display new information about availability zones: In the Overview tab, you can see Zones as an available Scope . In Network Traffic Traffic flows , Zones are viewable under the SrcK8S_Zone and DstK8S_Zone fields. In the Topology view, you can set Zones as Scope or Group . 7.2.7. Filtering eBPF flow data using a global rule You can configure the FlowCollector to filter eBPF flows using a global rule to control the flow of packets cached in the eBPF flow table. Procedure In the web console, navigate to Operators Installed Operators . Under the Provided APIs heading for Network Observability , select Flow Collector . Select cluster , then select the YAML tab. Configure the FlowCollector custom resource, similar to the following sample configurations: Example 7.1. Filter Kubernetes service traffic to a specific Pod IP endpoint apiVersion: flows.netobserv.io/v1beta2 kind: FlowCollector metadata: name: cluster spec: namespace: netobserv deploymentModel: Direct agent: type: eBPF ebpf: flowFilter: action: Accept 1 cidr: 172.210.150.1/24 2 protocol: SCTP direction: Ingress destPortRange: 80-100 peerIP: 10.10.10.10 enable: true 3 1 The required action parameter describes the action that is taken for the flow filter rule. Possible values are Accept or Reject . 2 The required cidr parameter provides the IP address and CIDR mask for the flow filter rule and supports IPv4 and IPv6 address formats. If you want to match against any IP address, you can use 0.0.0.0/0 for IPv4 or ::/0 for IPv6. 3 You must set spec.agent.ebpf.flowFilter.enable to true to enable this feature. Example 7.2. See flows to any addresses outside the cluster apiVersion: flows.netobserv.io/v1beta2 kind: FlowCollector metadata: name: cluster spec: namespace: netobserv deploymentModel: Direct agent: type: eBPF ebpf: flowFilter: action: Accept 1 cidr: 0.0.0.0/0 2 protocol: TCP direction: Egress sourcePort: 100 peerIP: 192.168.127.12 3 enable: true 4 1 You can Accept flows based on the criteria in the flowFilter specification. 2 The cidr value of 0.0.0.0/0 matches against any IP address. 3 See flows after peerIP is configured with 192.168.127.12 . 4 You must set spec.agent.ebpf.flowFilter.enable to true to enable the feature. 7.2.8. Endpoint translation (xlat) You can gain visibility into the endpoints serving traffic in a consolidated view using Network Observability and extended Berkeley Packet Filter (eBPF). Typically, when traffic flows through a service, egressIP, or load balancer, the traffic flow information is abstracted as it is routed to one of the available pods. If you try to get information about the traffic, you can only view service related info, such as service IP and port, and not information about the specific pod that is serving the request. Often the information for both the service traffic and the virtual service endpoint is captured as two separate flows, which complicates troubleshooting. To solve this, endpoint xlat can help in the following ways: Capture the network flows at the kernel level, which has a minimal impact on performance. Enrich the network flows with translated endpoint information, showing not only the service but also the specific backend pod, so you can see which pod served a request. As network packets are processed, the eBPF hook enriches flow logs with metadata about the translated endpoint that includes the following pieces of information that you can view in the Network Traffic page in a single row: Source Pod IP Source Port Destination Pod IP Destination Port Conntrack Zone ID 7.2.9. Working with endpoint translation (xlat) You can use Network Observability and eBPF to enrich network flows from a Kubernetes service with translated endpoint information, gaining insight into the endpoints serving traffic. Procedure In the web console, navigate to Operators Installed Operators . In the Provided APIs heading for the NetObserv Operator , select Flow Collector . Select cluster , and then select the YAML tab. Configure the FlowCollector custom resource for PacketTranslation , for example: Example FlowCollector configuration apiVersion: flows.netobserv.io/v1beta2 kind: FlowCollector metadata: name: cluster spec: namespace: netobserv agent: type: eBPF ebpf: features: - PacketTranslation 1 1 You can start enriching network flows with translated packet information by listing the PacketTranslation parameter in the spec.agent.ebpf.features specification list. Example filtering When you refresh the Network Traffic page you can filter for information about translated packets: Filter the network flow data based on Destination kind: Service . You can see the xlat column, which distinguishes where translated information is displayed, and the following default columns: Xlat Zone ID Xlat Src Kubernetes Object Xlat Dst Kubernetes Object You can manage the display of additional xlat columns in Manage columns . 7.3. Observing the network traffic from the Topology view The Topology view provides a graphical representation of the network flows and the amount of traffic. As an administrator, you can monitor the traffic data across the application by using the Topology view. 7.3.1. Working with the Topology view As an administrator, you can navigate to the Topology view to see the details and metrics of the component. Procedure Navigate to Observe Network Traffic . In the Network Traffic page, click the Topology tab. You can click each component in the Topology to view the details and metrics of the component. 7.3.2. Configuring the advanced options for the Topology view You can customize and export the view by using Show advanced options . The advanced options view has the following features: Find in view : To search the required components in the view. Display options : To configure the following options: Edge labels : To show the specified measurements as edge labels. The default is to show the Average rate in Bytes . Scope : To select the scope of components between which the network traffic flows. The default value is Namespace . Groups : To enhance the understanding of ownership by grouping the components. The default value is None . Layout : To select the layout of the graphical representation. The default value is ColaNoForce . Show : To select the details that need to be displayed. All the options are checked by default. The options available are: Edges , Edges label , and Badges . Truncate labels : To select the required width of the label from the drop-down list. The default value is M . Collapse groups : To expand or collapse the groups. The groups are expanded by default. This option is disabled if Groups has the value of None . 7.3.2.1. Exporting the topology view To export the view, click Export topology view . The view is downloaded in PNG format. 7.4. Filtering the network traffic By default, the Network Traffic page displays the traffic flow data in the cluster based on the default filters configured in the FlowCollector instance. You can use the filter options to observe the required data by changing the preset filter. Query Options You can use Query Options to optimize the search results, as listed below: Log Type : The available options Conversation and Flows provide the ability to query flows by log type, such as flow log, new conversation, completed conversation, and a heartbeat, which is a periodic record with updates for long conversations. A conversation is an aggregation of flows between the same peers. Match filters : You can determine the relation between different filter parameters selected in the advanced filter. The available options are Match all and Match any . Match all provides results that match all the values, and Match any provides results that match any of the values entered. The default value is Match all . Datasource : You can choose the datasource to use for queries: Loki , Prometheus , or Auto . Notable performance improvements can be realized when using Prometheus as a datasource rather than Loki, but Prometheus supports a limited set of filters and aggregations. The default datasource is Auto , which uses Prometheus on supported queries or uses Loki if the query does not support Prometheus. Drops filter : You can view different levels of dropped packets with the following query options: Fully dropped shows flow records with fully dropped packets. Containing drops shows flow records that contain drops but can be sent. Without drops shows records that contain sent packets. All shows all the aforementioned records. Limit : The data limit for internal backend queries. Depending upon the matching and the filter settings, the number of traffic flow data is displayed within the specified limit. Quick filters The default values in Quick filters drop-down menu are defined in the FlowCollector configuration. You can modify the options from console. Advanced filters You can set the advanced filters, Common , Source , or Destination , by selecting the parameter to be filtered from the dropdown list. The flow data is filtered based on the selection. To enable or disable the applied filter, you can click on the applied filter listed below the filter options. You can toggle between One way and Back and forth filtering. The One way filter shows only Source and Destination traffic according to your filter selections. You can use Swap to change the directional view of the Source and Destination traffic. The Back and forth filter includes return traffic with the Source and Destination filters. The directional flow of network traffic is shown in the Direction column in the Traffic flows table as Ingress`or `Egress for inter-node traffic and `Inner`for traffic inside a single node. You can click Reset defaults to remove the existing filters, and apply the filter defined in FlowCollector configuration. Note To understand the rules of specifying the text value, click Learn More . Alternatively, you can access the traffic flow data in the Network Traffic tab of the Namespaces , Services , Routes , Nodes , and Workloads pages which provide the filtered data of the corresponding aggregations. Additional resources Configuring Quick Filters Flow Collector sample resource

Commands:
apiVersion: flows.netobserv.io/v1beta2 kind: FlowCollector metadata: name: cluster spec: processor: logTypes: Flows 1 advanced: conversationEndTimeout: 10s 2 conversationHeartbeatInterval: 30s 3
apiVersion: flows.netobserv.io/v1beta2 kind: FlowCollector metadata: name: cluster spec: namespace: netobserv agent: type: eBPF ebpf: features: - DNSTracking 1 sampling: 1 2
apiVersion: flows.netobserv.io/v1beta2 kind: FlowCollector metadata: name: cluster spec: namespace: netobserv agent: type: eBPF ebpf: features: - FlowRTT 1
apiVersion: flows.netobserv.io/v1beta2 kind: FlowCollector metadata: name: cluster spec: processor: addZone: true
apiVersion: flows.netobserv.io/v1beta2 kind: FlowCollector metadata: name: cluster spec: namespace: netobserv deploymentModel: Direct agent: type: eBPF ebpf: flowFilter: action: Accept 1 cidr: 172.210.150.1/24 2 protocol: SCTP direction: Ingress destPortRange: 80-100 peerIP: 10.10.10.10 enable: true 3
apiVersion: flows.netobserv.io/v1beta2 kind: FlowCollector metadata: name: cluster spec: namespace: netobserv deploymentModel: Direct agent: type: eBPF ebpf: flowFilter: action: Accept 1 cidr: 0.0.0.0/0 2 protocol: TCP direction: Egress sourcePort: 100 peerIP: 192.168.127.12 3 enable: true 4
apiVersion: flows.netobserv.io/v1beta2 kind: FlowCollector metadata: name: cluster spec: namespace: netobserv agent: type: eBPF ebpf: features: - PacketTranslation 1