Chapter 41. Monitoring disk and I/O activity with SystemTap You can monitor disk and I/O activity with the following scripts. 41.1. Summarizing disk read/write traffic with SystemTap You can use the disktop.stp SystemTap script to identify which processes are performing the heaviest disk reads and writes to the system. Prerequisites You have installed SystemTap as described in Installing Systemtap . Procedure Run the disktop.stp script: The script displays the top ten processes responsible for the heaviest reads or writes to a disk. The output includes the following data per listed process: UID User ID. A user ID of 0 refers to the root user. PID The ID of the listed process. PPID The process ID of the listed process's parent process. CMD The name of the listed process. DEVICE Which storage device the listed process is reading from or writing to. T The type of action performed by the listed process, where W refers to write, and R refers to read. BYTES The amount of data read to or written from disk. Output of the disktop.stp script looks similar to the following: 41.2. Tracking I/O time for each file read or write with SystemTap You can use the iotime.stp SystemTap script to monitor the amount of time it takes for each process to read from or write to any file. This helps you to determine what files are slow to load on a system. Prerequisites You have installed SystemTap as described in Installing Systemtap . Procedure Run the iotime.stp script: The script tracks each time a system call opens, closes, reads from, and writes to a file. For each file any system call accesses, It counts the number of microseconds it takes for any reads or writes to finish and tracks the amount of data , in bytes, read from or written to the file. The output contains: A timestamp, in microseconds Process ID and process name An access or iotime flag The file accessed If a process was able to read or write any data, a pair of access and iotime lines should appear together. The access line refers to the time that a given process started accessing a file. The end of the access line will show the amount of data read or written. The iotime line will show the amount of time, in microseconds, that the process took in order to perform the read or write. Output of the iotime.stp script looks similar to the following: 41.3. Tracking cumulative I/O with SystemTap You can use the traceio.stp SystemTap script to track the cumulative amount of I/O to the system. Prerequisites You have installed SystemTap as described in Installing Systemtap . Procedure Run the traceio.stp script: The script prints the top ten executables generating I/O traffic over time. It also tracks the cumulative amount of I/O reads and writes done by those executables. This information is tracked and printed out in 1-second intervals, and in descending order. Output of the traceio.stp script looks similar to the following: 41.4. Monitoring I/O activity on a specific device with SystemTap You can use the traceio2.stp SystemTap script to monitor I/O activity on a specific device. Prerequisites You have installed SystemTap as described in Installing Systemtap . Procedure Run the traceio2.stp script. This script takes the whole device number as an argument. To find this number you can use: Where directory is located on the device you want to monitor. The output contains following: The name and ID of any process performing a read or write The function it is performing ( vfs_read or vfs_write ) The kernel device number Consider following output of # stap traceio2.stp 0x805 41.5. Monitoring reads and writes to a file with SystemTap You can use the inodewatch.stp SystemTap script to monitor reads from and writes to a file in real time. Prerequisites You have installed SystemTap as described in Installing Systemtap . Procedure Run the inodewatch.stp script. The script inodewatch.stp takes three command-line arguments: The file's major device number. The file's minor device number. The file's inode number. You can get these numbers using: Where filename is an absolute path. Consider following example: The output should look like: where: 805 is the base-16 (hexadecimal) device number. The last two digits are the minor device number, and the remaining digits are the major number. 1078319 is the inode number. To start monitoring /etc/crontab , run: In the first two arguments you must use 0x prefixes for base-16 numbers. The output contains following: The name and ID of any process performing a read or write The function it is performing ( vfs_read or vfs_write ) The kernel device number The output of this example should look like:

Commands:
stap --example disktop.stp
[...] Mon Sep 29 03:38:28 2008 , Average: 19Kb/sec, Read: 7Kb, Write: 89Kb UID PID PPID CMD DEVICE T BYTES 0 26319 26294 firefox sda5 W 90229 0 2758 2757 pam_timestamp_c sda5 R 8064 0 2885 1 cupsd sda5 W 1678 Mon Sep 29 03:38:38 2008 , Average: 1Kb/sec, Read: 7Kb, Write: 1Kb UID PID PPID CMD DEVICE T BYTES 0 2758 2757 pam_timestamp_c sda5 R 8064 0 2885 1 cupsd sda5 W 1678
stap --example iotime.stp
[...] 825946 3364 (NetworkManager) access /sys/class/net/eth0/carrier read: 8190 write: 0 825955 3364 (NetworkManager) iotime /sys/class/net/eth0/carrier time: 9 [...] 117061 2460 (pcscd) access /dev/bus/usb/003/001 read: 43 write: 0 117065 2460 (pcscd) iotime /dev/bus/usb/003/001 time: 7 [...] 3973737 2886 (sendmail) access /proc/loadavg read: 4096 write: 0 3973744 2886 (sendmail) iotime /proc/loadavg time: 11 [...]
stap --example traceio.stp
[...] Xorg r: 583401 KiB w: 0 KiB floaters r: 96 KiB w: 7130 KiB multiload-apple r: 538 KiB w: 537 KiB sshd r: 71 KiB w: 72 KiB pam_timestamp_c r: 138 KiB w: 0 KiB staprun r: 51 KiB w: 51 KiB snmpd r: 46 KiB w: 0 KiB pcscd r: 28 KiB w: 0 KiB irqbalance r: 27 KiB w: 4 KiB cupsd r: 4 KiB w: 18 KiB Xorg r: 588140 KiB w: 0 KiB floaters r: 97 KiB w: 7143 KiB multiload-apple r: 543 KiB w: 542 KiB sshd r: 72 KiB w: 72 KiB pam_timestamp_c r: 138 KiB w: 0 KiB staprun r: 51 KiB w: 51 KiB snmpd r: 46 KiB w: 0 KiB pcscd r: 28 KiB w: 0 KiB irqbalance r: 27 KiB w: 4 KiB cupsd r: 4 KiB w: 18 KiB
stap --example traceio2.stp 'argument'
stat -c "0x%D" directory
[...] synergyc(3722) vfs_read 0x800005 synergyc(3722) vfs_read 0x800005 cupsd(2889) vfs_write 0x800005 cupsd(2889) vfs_write 0x800005 cupsd(2889) vfs_write 0x800005 [...]
stap --example inodewatch.stp 'argument1' 'argument2' 'argument3'
stat -c '%D %i' filename
stat -c '%D %i' /etc/crontab
805 1078319
stap inodewatch.stp 0x8 0x05 1078319
cat(16437) vfs_read 0x800005/1078319 cat(16437) vfs_read 0x800005/1078319