Chapter 4. Useful SystemTap Scripts This chapter enumerates several SystemTap scripts you can use to monitor and investigate different subsystems. All of these scripts are available in the /usr/share/systemtap/testsuite/systemtap.examples/ directory once you install the systemtap-testsuite package. 4.1. Network The following sections showcase scripts that trace network-related functions and build a profile of network activity. 4.1.1. Network Profiling This section describes how to profile network activity. Example 4.1, "nettop.stp" provides a glimpse into how much network traffic each process is generating on a machine. Example 4.1. nettop.stp #! /usr/bin/env stap global ifxmit, ifrecv global ifmerged probe netdev.transmit { ifxmit[pid(), dev_name, execname(), uid()] <<< length } probe netdev.receive { ifrecv[pid(), dev_name, execname(), uid()] <<< length } function print_activity() { printf("%5s %5s %-7s %7s %7s %7s %7s %-15s\n", "PID", "UID", "DEV", "XMIT_PK", "RECV_PK", "XMIT_KB", "RECV_KB", "COMMAND") foreach ([pid, dev, exec, uid] in ifrecv) { ifmerged[pid, dev, exec, uid] += @count(ifrecv[pid,dev,exec,uid]); } foreach ([pid, dev, exec, uid] in ifxmit) { ifmerged[pid, dev, exec, uid] += @count(ifxmit[pid,dev,exec,uid]); } foreach ([pid, dev, exec, uid] in ifmerged-) { n_xmit = @count(ifxmit[pid, dev, exec, uid]) n_recv = @count(ifrecv[pid, dev, exec, uid]) printf("%5d %5d %-7s %7d %7d %7d %7d %-15s\n", pid, uid, dev, n_xmit, n_recv, n_xmit ? @sum(ifxmit[pid, dev, exec, uid])/1024 : 0, n_recv ? @sum(ifrecv[pid, dev, exec, uid])/1024 : 0, exec) } print("\n") delete ifxmit delete ifrecv delete ifmerged } probe timer.ms(5000), end, error { print_activity() } Note that the print_activity() function uses the following expressions: n_xmit ? @sum(ifxmit[pid, dev, exec, uid])/1024 : 0 n_recv ? @sum(ifrecv[pid, dev, exec, uid])/1024 : 0 These expressions are if or else conditionals. The second statement is simply a more concise way of writing the following pseudo code: if n_recv != 0 then @sum(ifrecv[pid, dev, exec, uid])/1024 else 0 Example 4.1, "nettop.stp" tracks which processes are generating network traffic on the system, and provides the following information about each process: PID - the ID of the listed process. UID - user ID. A user ID of 0 refers to the root user. DEV - which ethernet device the process used to send or receive data (for example, eth0, eth1) XMIT_PK - number of packets transmitted by the process RECV_PK - number of packets received by the process XMIT_KB - amount of data sent by the process, in kilobytes RECV_KB - amount of data received by the service, in kilobytes Example 4.1, "nettop.stp" provides network profile sampling every 5 seconds. You can change this setting by editing probe timer.ms(5000) accordingly. Example 4.2, "Example 4.1, "nettop.stp" Sample Output" contains an excerpt of the output from Example 4.1, "nettop.stp" over a 20-second period: Example 4.2. Example 4.1, "nettop.stp" Sample Output 4.1.2. Tracing Functions Called in Network Socket Code This section describes how to trace functions called from the kernel's net/socket.c file. This task helps you identify, in finer detail, how each process interacts with the network at the kernel level. Example 4.3. socket-trace.stp #!/usr/bin/stap probe kernel.function("*@net/socket.c").call { printf ("%s -> %s\n", thread_indent(1), probefunc()) } probe kernel.function("*@net/socket.c").return { printf ("%s <- %s\n", thread_indent(-1), probefunc()) } Example 4.3, "socket-trace.stp" is identical to Example 3.6, "thread_indent.stp" , which was earlier used in SystemTap Functions to illustrate how thread_indent() works. Example 4.4. Example 4.3, "socket-trace.stp" Sample Output Example 4.4, "Example 4.3, "socket-trace.stp" Sample Output" contains a 3-second excerpt of the output for Example 4.3, "socket-trace.stp" . For more information about the output of this script as provided by thread_indent() , see SystemTap Functions Example 3.6, "thread_indent.stp" . 4.1.3. Monitoring Incoming TCP Connections This section illustrates how to monitor incoming TCP connections. This task is useful in identifying any unauthorized, suspicious, or otherwise unwanted network access requests in real time. Example 4.5. tcp_connections.stp #! /usr/bin/env stap probe begin { printf("%6s %16s %6s %6s %16s\n", "UID", "CMD", "PID", "PORT", "IP_SOURCE") } probe kernel.function("tcp_accept").return?, kernel.function("inet_csk_accept").return? { sock = USDreturn if (sock != 0) printf("%6d %16s %6d %6d %16s\n", uid(), execname(), pid(), inet_get_local_port(sock), inet_get_ip_source(sock)) } While Example 4.5, "tcp_connections.stp" is running, it will print out the following information about any incoming TCP connections accepted by the system in real time: Current UID CMD - the command accepting the connection PID of the command Port used by the connection IP address from which the TCP connection originated Example 4.6. Example 4.5, "tcp_connections.stp" Sample Output 4.1.4. Monitoring Network Packets Drops in Kernel The network stack in Linux can discard packets for various reasons. Some Linux kernels include a tracepoint, kernel.trace("kfree_skb") , which easily tracks where packets are discarded. Example 4.7, "dropwatch.stp" uses kernel.trace("kfree_skb") to trace packet discards; the script summarizes which locations discard packets every five-second interval. Example 4.7. dropwatch.stp #!/usr/bin/stap ############################################################ # Dropwatch.stp # Author: Neil Horman <nhorman@redhat.com> # An example script to mimic the behavior of the dropwatch utility # http://fedorahosted.org/dropwatch ############################################################ # Array to hold the list of drop points we find global locations # Note when we turn the monitor on and off probe begin { printf("Monitoring for dropped packets\n") } probe end { printf("Stopping dropped packet monitor\n") } # increment a drop counter for every location we drop at probe kernel.trace("kfree_skb") { locations[USDlocation] <<< 1 } # Every 5 seconds report our drop locations probe timer.sec(5) { printf("\n") foreach (l in locations-) { printf("%d packets dropped at location %p\n", @count(locations[l]), l) } delete locations } The kernel.trace("kfree_skb") traces which places in the kernel drop network packets. The kernel.trace("kfree_skb") has two arguments: a pointer to the buffer being freed ( USDskb ) and the location in kernel code the buffer is being freed ( USDlocation ). Running the dropwatch.stp script 15 seconds would result in output similar in Example 4.8, "Example 4.7, "dropwatch.stp" Sample Output" . The output lists the number of misses for tracepoint address and the actual address. Example 4.8. Example 4.7, "dropwatch.stp" Sample Output To make the location of packet drops more meaningful, see the /boot/System.map-USD(uname -r) file. This file lists the starting addresses for each function, allowing you to map the addresses in the output of Example 4.8, "Example 4.7, "dropwatch.stp" Sample Output" to a specific function name. Given the following snippet of the /boot/System.map-USD(uname -r) file, the address 0xffffffff8024cd0f maps to the function unix_stream_recvmsg and the address 0xffffffff8044b472 maps to the function arp_rcv :

Commands:
#! /usr/bin/env stap global ifxmit, ifrecv global ifmerged probe netdev.transmit { ifxmit[pid(), dev_name, execname(), uid()] <<< length } probe netdev.receive { ifrecv[pid(), dev_name, execname(), uid()] <<< length } function print_activity() { printf("%5s %5s %-7s %7s %7s %7s %7s %-15s\n", "PID", "UID", "DEV", "XMIT_PK", "RECV_PK", "XMIT_KB", "RECV_KB", "COMMAND") foreach ([pid, dev, exec, uid] in ifrecv) { ifmerged[pid, dev, exec, uid] += @count(ifrecv[pid,dev,exec,uid]); } foreach ([pid, dev, exec, uid] in ifxmit) { ifmerged[pid, dev, exec, uid] += @count(ifxmit[pid,dev,exec,uid]); } foreach ([pid, dev, exec, uid] in ifmerged-) { n_xmit = @count(ifxmit[pid, dev, exec, uid]) n_recv = @count(ifrecv[pid, dev, exec, uid]) printf("%5d %5d %-7s %7d %7d %7d %7d %-15s\n", pid, uid, dev, n_xmit, n_recv, n_xmit ? @sum(ifxmit[pid, dev, exec, uid])/1024 : 0, n_recv ? @sum(ifrecv[pid, dev, exec, uid])/1024 : 0, exec) } print("\n") delete ifxmit delete ifrecv delete ifmerged } probe timer.ms(5000), end, error { print_activity() }
n_xmit ? @sum(ifxmit[pid, dev, exec, uid])/1024 : 0 n_recv ? @sum(ifrecv[pid, dev, exec, uid])/1024 : 0
if n_recv != 0 then @sum(ifrecv[pid, dev, exec, uid])/1024 else 0
[...] PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND 0 0 eth0 0 5 0 0 swapper 11178 0 eth0 2 0 0 0 synergyc PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND 2886 4 eth0 79 0 5 0 cups-polld 11362 0 eth0 0 61 0 5 firefox 0 0 eth0 3 32 0 3 swapper 2886 4 lo 4 4 0 0 cups-polld 11178 0 eth0 3 0 0 0 synergyc PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND 0 0 eth0 0 6 0 0 swapper 2886 4 lo 2 2 0 0 cups-polld 11178 0 eth0 3 0 0 0 synergyc 3611 0 eth0 0 1 0 0 Xorg PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND 0 0 eth0 3 42 0 2 swapper 11178 0 eth0 43 1 3 0 synergyc 11362 0 eth0 0 7 0 0 firefox 3897 0 eth0 0 1 0 0 multiload-apple [...]
#!/usr/bin/stap probe kernel.function("*@net/socket.c").call { printf ("%s -> %s\n", thread_indent(1), probefunc()) } probe kernel.function("*@net/socket.c").return { printf ("%s <- %s\n", thread_indent(-1), probefunc()) }
[...] 0 Xorg(3611): -> sock_poll 3 Xorg(3611): <- sock_poll 0 Xorg(3611): -> sock_poll 3 Xorg(3611): <- sock_poll 0 gnome-terminal(11106): -> sock_poll 5 gnome-terminal(11106): <- sock_poll 0 scim-bridge(3883): -> sock_poll 3 scim-bridge(3883): <- sock_poll 0 scim-bridge(3883): -> sys_socketcall 4 scim-bridge(3883): -> sys_recv 8 scim-bridge(3883): -> sys_recvfrom 12 scim-bridge(3883):-> sock_from_file 16 scim-bridge(3883):<- sock_from_file 20 scim-bridge(3883):-> sock_recvmsg 24 scim-bridge(3883):<- sock_recvmsg 28 scim-bridge(3883): <- sys_recvfrom 31 scim-bridge(3883): <- sys_recv 35 scim-bridge(3883): <- sys_socketcall [...]
#! /usr/bin/env stap probe begin { printf("%6s %16s %6s %6s %16s\n", "UID", "CMD", "PID", "PORT", "IP_SOURCE") } probe kernel.function("tcp_accept").return?, kernel.function("inet_csk_accept").return? { sock = USDreturn if (sock != 0) printf("%6d %16s %6d %6d %16s\n", uid(), execname(), pid(), inet_get_local_port(sock), inet_get_ip_source(sock)) }
UID CMD PID PORT IP_SOURCE 0 sshd 3165 22 10.64.0.227 0 sshd 3165 22 10.64.0.227
#!/usr/bin/stap ############################################################ Dropwatch.stp Author: Neil Horman <nhorman@redhat.com> An example script to mimic the behavior of the dropwatch utility http://fedorahosted.org/dropwatch ############################################################ Array to hold the list of drop points we find global locations Note when we turn the monitor on and off probe begin { printf("Monitoring for dropped packets\n") } probe end { printf("Stopping dropped packet monitor\n") } increment a drop counter for every location we drop at probe kernel.trace("kfree_skb") { locations[USDlocation] <<< 1 } Every 5 seconds report our drop locations probe timer.sec(5) { printf("\n") foreach (l in locations-) { printf("%d packets dropped at location %p\n", @count(locations[l]), l) } delete locations }
Monitoring for dropped packets 51 packets dropped at location 0xffffffff8024cd0f 2 packets dropped at location 0xffffffff8044b472 51 packets dropped at location 0xffffffff8024cd0f 1 packets dropped at location 0xffffffff8044b472 97 packets dropped at location 0xffffffff8024cd0f 1 packets dropped at location 0xffffffff8044b472 Stopping dropped packet monitor
[...] ffffffff8024c5cd T unlock_new_inode ffffffff8024c5da t unix_stream_sendmsg ffffffff8024c920 t unix_stream_recvmsg ffffffff8024cea1 t udp_v4_lookup_longway [...] ffffffff8044addc t arp_process ffffffff8044b360 t arp_rcv ffffffff8044b487 t parp_redo ffffffff8044b48c t arp_solicit [...]