Chapter 39. Monitoring network activity with SystemTap You can use helpful example SystemTap scripts available in the /usr/share/systemtap/testsuite/systemtap.examples/ directory, upon installing the systemtap-testsuite package, to monitor and investigate the network activity of your system. 39.1. Profiling network activity with SystemTap You can use the nettop.stp example SystemTap script to profile network activity. The script tracks which processes are generating network traffic on the system, and provides the following information about each process: PID The ID of the listed process. UID User ID. A user ID of 0 refers to the root user. DEV Which ethernet device the process used to send or receive data (for example, eth0, eth1). XMIT_PK The number of packets transmitted by the process. RECV_PK The number of packets received by the process. XMIT_KB The amount of data sent by the process, in kilobytes. RECV_KB The amount of data received by the service, in kilobytes. Prerequisites You have installed SystemTap as described in Installing SystemTap . Procedure Run the nettop.stp script: The nettop.stp script provides network profile sampling every 5 seconds. Output of the nettop.stp script looks similar to the following: 39.2. Tracing functions called in network socket code with SystemTap You can use the socket-trace.stp example SystemTap script to trace functions called from the kernel's net/socket.c file. This helps you identify, in finer detail, how each process interacts with the network at the kernel level. Prerequisites You have installed SystemTap as described in Installing SystemTap . Procedure Run the socket-trace.stp script: A 3-second excerpt of the output of the socket-trace.stp script looks similar to the following: 39.3. Monitoring network packet drops with SystemTap The network stack in Linux can discard packets for various reasons. Some Linux kernels include a tracepoint, kernel.trace("kfree_skb") , which tracks where packets are discarded. The dropwatch.stp SystemTap script uses kernel.trace("kfree_skb") to trace packet discards; the script summarizes what locations discard packets in every 5-second interval. Prerequisites You have installed SystemTap as described in Installing SystemTap . Procedure Run the dropwatch.stp script: Running the dropwatch.stp script for 15 seconds results in output similar to the following: Note To make the location of packet drops more meaningful, see the /boot/System.map-USD(uname -r) file. This file lists the starting addresses for each function, enabling you to map the addresses in the output of the dropwatch.stp script to a specific function name. Given the following snippet of the /boot/System.map-USD(uname -r) file, the address 0xffffffff8024cd0f maps to the function unix_stream_recvmsg and the address 0xffffffff8044b472 maps to the function arp_rcv :

Commands:
stap --example nettop.stp
[...] PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND 0 0 eth0 0 5 0 0 swapper 11178 0 eth0 2 0 0 0 synergyc PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND 2886 4 eth0 79 0 5 0 cups-polld 11362 0 eth0 0 61 0 5 firefox 0 0 eth0 3 32 0 3 swapper 2886 4 lo 4 4 0 0 cups-polld 11178 0 eth0 3 0 0 0 synergyc PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND 0 0 eth0 0 6 0 0 swapper 2886 4 lo 2 2 0 0 cups-polld 11178 0 eth0 3 0 0 0 synergyc 3611 0 eth0 0 1 0 0 Xorg PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND 0 0 eth0 3 42 0 2 swapper 11178 0 eth0 43 1 3 0 synergyc 11362 0 eth0 0 7 0 0 firefox 3897 0 eth0 0 1 0 0 multiload-apple
stap --example socket-trace.stp
[...] 0 Xorg(3611): -> sock_poll 3 Xorg(3611): <- sock_poll 0 Xorg(3611): -> sock_poll 3 Xorg(3611): <- sock_poll 0 gnome-terminal(11106): -> sock_poll 5 gnome-terminal(11106): <- sock_poll 0 scim-bridge(3883): -> sock_poll 3 scim-bridge(3883): <- sock_poll 0 scim-bridge(3883): -> sys_socketcall 4 scim-bridge(3883): -> sys_recv 8 scim-bridge(3883): -> sys_recvfrom 12 scim-bridge(3883):-> sock_from_file 16 scim-bridge(3883):<- sock_from_file 20 scim-bridge(3883):-> sock_recvmsg 24 scim-bridge(3883):<- sock_recvmsg 28 scim-bridge(3883): <- sys_recvfrom 31 scim-bridge(3883): <- sys_recv 35 scim-bridge(3883): <- sys_socketcall [...]
stap --example dropwatch.stp
Monitoring for dropped packets 51 packets dropped at location 0xffffffff8024cd0f 2 packets dropped at location 0xffffffff8044b472 51 packets dropped at location 0xffffffff8024cd0f 1 packets dropped at location 0xffffffff8044b472 97 packets dropped at location 0xffffffff8024cd0f 1 packets dropped at location 0xffffffff8044b472 Stopping dropped packet monitor
[...] ffffffff8024c5cd T unlock_new_inode ffffffff8024c5da t unix_stream_sendmsg ffffffff8024c920 t unix_stream_recvmsg ffffffff8024cea1 t udp_v4_lookup_longway [...] ffffffff8044addc t arp_process ffffffff8044b360 t arp_rcv ffffffff8044b487 t parp_redo ffffffff8044b48c t arp_solicit [...]