Chapter 46. Network tracing using the BPF compiler collection BPF Compiler Collection (BCC) is a library, which facilitates the creation of the extended Berkeley Packet Filter (eBPF) programs. The main utility of eBPF programs is analyzing the operating system performance and network performance without experiencing overhead or security issues. BCC removes the need for users to know deep technical details of eBPF, and provides many out-of-the-box starting points, such as the bcc-tools package with pre-created eBPF programs. Note The eBPF programs are triggered on events, such as disk I/O, TCP connections, and process creations. It is unlikely that the programs should cause the kernel to crash, loop or become unresponsive because they run in a safe virtual machine in the kernel. 46.1. Installing the bcc-tools package Install the bcc-tools package, which also installs the BPF Compiler Collection (BCC) library as a dependency. Procedure Install bcc-tools . The BCC tools are installed in the /usr/share/bcc/tools/ directory. Verification Inspect the installed tools: The doc directory in the listing provides documentation for each tool. 46.2. Displaying TCP connections added to the Kernel's accept queue After the kernel receives the ACK packet in a TCP 3-way handshake, the kernel moves the connection from the SYN queue to the accept queue after the connection's state changes to ESTABLISHED . Therefore, only successful TCP connections are visible in this queue. The tcpaccept utility uses eBPF features to display all connections the kernel adds to the accept queue. The utility is lightweight because it traces the accept() function of the kernel instead of capturing packets and filtering them. For example, use tcpaccept for general troubleshooting to display new connections the server has accepted. Procedure Enter the following command to start the tracing the kernel accept queue: Each time the kernel accepts a connection, tcpaccept displays the details of the connections. Press Ctrl + C to stop the tracing process. Additional resources tcpaccept(8) man page on your system /usr/share/bcc/tools/doc/tcpaccept_example.txt file 46.3. Tracing outgoing TCP connection attempts The tcpconnect utility uses eBPF features to trace outgoing TCP connection attempts. The output of the utility also includes connections that failed. The tcpconnect utility is lightweight because it traces, for example, the connect() function of the kernel instead of capturing packets and filtering them. Procedure Enter the following command to start the tracing process that displays all outgoing connections: Each time the kernel processes an outgoing connection, tcpconnect displays the details of the connections. Press Ctrl + C to stop the tracing process. Additional resources tcpconnect(8) man page on your system /usr/share/bcc/tools/doc/tcpconnect_example.txt file 46.4. Measuring the latency of outgoing TCP connections The TCP connection latency is the time taken to establish a connection. This typically involves the kernel TCP/IP processing and network round trip time, and not the application runtime. The tcpconnlat utility uses eBPF features to measure the time between a sent SYN packet and the received response packet. Procedure Start measuring the latency of outgoing connections: Each time the kernel processes an outgoing connection, tcpconnlat displays the details of the connection after the kernel receives the response packet. Press Ctrl + C to stop the tracing process. Additional resources tcpconnlat(8) man page on your system /usr/share/bcc/tools/doc/tcpconnlat_example.txt file 46.5. Displaying details about TCP packets and segments that were dropped by the kernel The tcpdrop utility enables administrators to display details about TCP packets and segments that were dropped by the kernel. Use this utility to debug high rates of dropped packets that can cause the remote system to send timer-based retransmits. High rates of dropped packets and segments can impact the performance of a server. Instead of capturing and filtering packets, which is resource-intensive, the tcpdrop utility uses eBPF features to retrieve the information directly from the kernel. Procedure Enter the following command to start displaying details about dropped TCP packets and segments: Each time the kernel drops TCP packets and segments, tcpdrop displays the details of the connection, including the kernel stack trace that led to the dropped package. Press Ctrl + C to stop the tracing process. Additional resources tcpdrop(8) man page on your system /usr/share/bcc/tools/doc/tcpdrop_example.txt file 46.6. Tracing TCP sessions The tcplife utility uses eBPF to trace TCP sessions that open and close, and prints a line of output to summarize each one. Administrators can use tcplife to identify connections and the amount of transferred traffic. For example, you can display connections to port 22 (SSH) to retrieve the following information: The local process ID (PID) The local process name The local IP address and port number The remote IP address and port number The amount of received and transmitted traffic in KB. The time in milliseconds the connection was active Procedure Enter the following command to start the tracing of connections to the local port 22 : Each time a connection is closed, tcplife displays the details of the connections. Press Ctrl + C to stop the tracing process. Additional resources tcplife(8) man page on your system /usr/share/bcc/tools/doc/tcplife_example.txt file 46.7. Tracing TCP retransmissions The tcpretrans utility displays details about TCP retransmissions, such as the local and remote IP address and port number, as well as the TCP state at the time of the retransmissions. The utility uses eBPF features and, therefore, has a very low overhead. Procedure Use the following command to start displaying TCP retransmission details: Each time the kernel calls the TCP retransmit function, tcpretrans displays the details of the connection. Press Ctrl + C to stop the tracing process. Additional resources tcpretrans(8) man page on your system /usr/share/bcc/tools/doc/tcpretrans_example.txt file 46.8. Displaying TCP state change information During a TCP session, the TCP state changes. The tcpstates utility uses eBPF functions to trace these state changes, and prints details including the duration in each state. For example, use tcpstates to identify if connections spend too much time in the initialization state. Procedure Use the following command to start tracing TCP state changes: Each time a connection changes its state, tcpstates displays a new line with updated connection details. If multiple connections change their state at the same time, use the socket address in the first column ( SKADDR ) to determine which entries belong to the same connection. Press Ctrl + C to stop the tracing process. Additional resources tcpstates(8) man page on your system /usr/share/bcc/tools/doc/tcpstates_example.txt file 46.9. Summarizing and aggregating TCP traffic sent to specific subnets The tcpsubnet utility summarizes and aggregates IPv4 TCP traffic that the local host sends to subnets and displays the output on a fixed interval. The utility uses eBPF features to collect and summarize the data to reduce the overhead. By default, tcpsubnet summarizes traffic for the following subnets: 127.0.0.1/32 10.0.0.0/8 172.16.0.0/12 192.0.2.0/24/16 0.0.0.0/0 Note that the last subnet ( 0.0.0.0/0 ) is a catch-all option. The tcpsubnet utility counts all traffic for subnets different than the first four in this catch-all entry. Follow the procedure to count the traffic for the 192.0.2.0/24 and 198.51.100.0/24 subnets. Traffic to other subnets will be tracked in the 0.0.0.0/0 catch-all subnet entry. Procedure Start monitoring the amount of traffic sent to the 192.0.2.0/24 , 198.51.100.0/24 , and other subnets: This command displays the traffic in bytes for the specified subnets once per second. Press Ctrl + C to stop the tracing process. Additional resources tcpsubnet(8) man page on your system /usr/share/bcc/tools/doc/tcpsubnet.txt file 46.10. Displaying the network throughput by IP address and port The tcptop utility displays TCP traffic the host sends and receives in kilobytes. The report automatically refreshes and contains only active TCP connections. The utility uses eBPF features and, therefore, has only a very low overhead. Procedure To monitor the sent and received traffic, enter: The output of the command includes only active TCP connections. If the local or remote system closes a connection, the connection is no longer visible in the output. Press Ctrl + C to stop the tracing process. Additional resources tcptop(8) man page on your system /usr/share/bcc/tools/doc/tcptop.txt file 46.11. Tracing established TCP connections The tcptracer utility traces the kernel functions that connect, accept, and close TCP connections. The utility uses eBPF features and, therefore, has a very low overhead. Procedure Use the following command to start the tracing process: Each time the kernel connects, accepts, or closes a connection, tcptracer displays the details of the connections. Press Ctrl + C to stop the tracing process. Additional resources tcptracer(8) man page on your system /usr/share/bcc/tools/doc/tcptracer_example.txt file 46.12. Tracing IPv4 and IPv6 listen attempts The solisten utility traces all IPv4 and IPv6 listen attempts. It traces the attempts including which ultimately fail or the listening program that does not accept the connection. The utility traces functions that the kernel calls when a program wants to listen for TCP connections. Procedure Enter the following command to start the tracing process that displays all listen TCP attempts: Press Ctrl + C to stop the tracing process. Additional resources solisten(9) man page on your system /usr/share/bcc/tools/doc/solisten_example.txt file 46.13. Summarizing the service time of soft interrupts The softirqs utility summarizes the time spent servicing soft interrupts (soft IRQs) and shows this time as either totals or histogram distributions. This utility uses the irq:softirq_enter and irq:softirq_exit kernel tracepoints, which is a stable tracing mechanism. Procedure Enter the following command to start the tracing soft irq event time: Press Ctrl + C to stop the tracing process. Additional resources softirqs(8) and mpstat(1) man pages on your system /usr/share/bcc/tools/doc/softirqs_example.txt file 46.14. Summarizing packets size and count on a network interface The netqtop utility displays statistics about the attributes of received (RX) and transmitted (TX) packets on each network queue of a particular network interface. The statistics include: Bytes per second (BPS) Packets per second (PPS) The average packet size Total number of packets To generate these statistics, netqtop traces the kernel functions that perform events of transmitted packets net_dev_start_xmit and received packets netif_receive_skb . Procedure Display the number of packets within the range of bytes size of the time interval of 2 seconds: Press Ctrl + C to stop netqtop . Additional resources netqtop(8) man page on your system /usr/share/bcc/tools/doc/netqtop_example.txt

Commands:
yum install bcc-tools
ls -l /usr/share/bcc/tools/ -rwxr-xr-x. 1 root root 4198 Dec 14 17:53 dcsnoop -rwxr-xr-x. 1 root root 3931 Dec 14 17:53 dcstat -rwxr-xr-x. 1 root root 20040 Dec 14 17:53 deadlock_detector -rw-r--r--. 1 root root 7105 Dec 14 17:53 deadlock_detector.c drwxr-xr-x. 3 root root 8192 Mar 11 10:28 doc -rwxr-xr-x. 1 root root 7588 Dec 14 17:53 execsnoop -rwxr-xr-x. 1 root root 6373 Dec 14 17:53 ext4dist -rwxr-xr-x. 1 root root 10401 Dec 14 17:53 ext4slower
/usr/share/bcc/tools/tcpaccept PID COMM IP RADDR RPORT LADDR LPORT 843 sshd 4 192.0.2.17 50598 192.0.2.1 22 1107 ns-slapd 4 198.51.100.6 38772 192.0.2.1 389 1107 ns-slapd 4 203.0.113.85 38774 192.0.2.1 389
/usr/share/bcc/tools/tcpconnect PID COMM IP SADDR DADDR DPORT 31346 curl 4 192.0.2.1 198.51.100.16 80 31348 telnet 4 192.0.2.1 203.0.113.231 23 31361 isc-worker00 4 192.0.2.1 192.0.2.254 53
/usr/share/bcc/tools/tcpconnlat PID COMM IP SADDR DADDR DPORT LAT(ms) 32151 isc-worker00 4 192.0.2.1 192.0.2.254 53 0.60 32155 ssh 4 192.0.2.1 203.0.113.190 22 26.34 32319 curl 4 192.0.2.1 198.51.100.59 443 188.96
/usr/share/bcc/tools/tcpdrop TIME PID IP SADDR:SPORT > DADDR:DPORT STATE (FLAGS) 13:28:39 32253 4 192.0.2.85:51616 > 192.0.2.1:22 CLOSE_WAIT (FIN|ACK) b'tcp_drop+0x1' b'tcp_data_queue+0x2b9' 13:28:39 1 4 192.0.2.85:51616 > 192.0.2.1:22 CLOSE (ACK) b'tcp_drop+0x1' b'tcp_rcv_state_process+0xe2'
/usr/share/bcc/tools/tcplife -L 22 PID COMM LADDR LPORT RADDR RPORT TX_KB RX_KB MS 19392 sshd 192.0.2.1 22 192.0.2.17 43892 53 52 6681.95 19431 sshd 192.0.2.1 22 192.0.2.245 43902 81 249381 7585.09 19487 sshd 192.0.2.1 22 192.0.2.121 43970 6998 7 16740.35
/usr/share/bcc/tools/tcpretrans TIME PID IP LADDR:LPORT T> RADDR:RPORT STATE 00:23:02 0 4 192.0.2.1:22 R> 198.51.100.0:26788 ESTABLISHED 00:23:02 0 4 192.0.2.1:22 R> 198.51.100.0:26788 ESTABLISHED 00:45:43 0 4 192.0.2.1:22 R> 198.51.100.0:17634 ESTABLISHED
/usr/share/bcc/tools/tcpstates SKADDR C-PID C-COMM LADDR LPORT RADDR RPORT OLDSTATE -> NEWSTATE MS ffff9cd377b3af80 0 swapper/1 0.0.0.0 22 0.0.0.0 0 LISTEN -> SYN_RECV 0.000 ffff9cd377b3af80 0 swapper/1 192.0.2.1 22 192.0.2.45 53152 SYN_RECV -> ESTABLISHED 0.067 ffff9cd377b3af80 818 sssd_nss 192.0.2.1 22 192.0.2.45 53152 ESTABLISHED -> CLOSE_WAIT 65636.773 ffff9cd377b3af80 1432 sshd 192.0.2.1 22 192.0.2.45 53152 CLOSE_WAIT -> LAST_ACK 24.409 ffff9cd377b3af80 1267 pulseaudio 192.0.2.1 22 192.0.2.45 53152 LAST_ACK -> CLOSE 0.376
/usr/share/bcc/tools/tcpsubnet 192.0.2.0/24,198.51.100.0/24,0.0.0.0/0 Tracing... Output every 1 secs. Hit Ctrl-C to end [02/21/20 10:04:50] 192.0.2.0/24 856 198.51.100.0/24 7467 [02/21/20 10:04:51] 192.0.2.0/24 1200 198.51.100.0/24 8763 0.0.0.0/0 673
/usr/share/bcc/tools/tcptop 13:46:29 loadavg: 0.10 0.03 0.01 1/215 3875 PID COMM LADDR RADDR RX_KB TX_KB 3853 3853 192.0.2.1:22 192.0.2.165:41838 32 102626 1285 sshd 192.0.2.1:22 192.0.2.45:39240 0 0
/usr/share/bcc/tools/tcptracer Tracing TCP established connections. Ctrl-C to end. T PID COMM IP SADDR DADDR SPORT DPORT A 1088 ns-slapd 4 192.0.2.153 192.0.2.1 0 65535 A 845 sshd 4 192.0.2.1 192.0.2.67 22 42302 X 4502 sshd 4 192.0.2.1 192.0.2.67 22 42302
/usr/share/bcc/tools/solisten PID COMM PROTO BACKLOG PORT ADDR 3643 nc TCPv4 1 4242 0.0.0.0 3659 nc TCPv6 1 4242 2001:db8:1::1 4221 redis-server TCPv6 128 6379 :: 4221 redis-server TCPv4 128 6379 0.0.0.0 .
/usr/share/bcc/tools/softirqs Tracing soft irq event time... Hit Ctrl-C to end. ^C SOFTIRQ TOTAL_usecs tasklet 166 block 9152 net_rx 12829 rcu 53140 sched 182360 timer 306256
/usr/share/bcc/tools/netqtop -n enp1s0 -i 2 Fri Jan 31 18:08:55 2023 TX QueueID avg_size [0, 64) [64, 512) [512, 2K) [2K, 16K) [16K, 64K) 0 0 0 0 0 0 0 Total 0 0 0 0 0 0 RX QueueID avg_size [0, 64) [64, 512) [512, 2K) [2K, 16K) [16K, 64K) 0 38.0 1 0 0 0 0 Total 38.0 1 0 0 0 0 ----------------------------------------------------------------------------- Fri Jan 31 18:08:57 2023 TX QueueID avg_size [0, 64) [64, 512) [512, 2K) [2K, 16K) [16K, 64K) 0 0 0 0 0 0 0 Total 0 0 0 0 0 0 RX QueueID avg_size [0, 64) [64, 512) [512, 2K) [2K, 16K) [16K, 64K) 0 38.0 1 0 0 0 0 Total 38.0 1 0 0 0 0 -----------------------------------------------------------------------------