4.2.2. Tracking I/O Time For Each File Read or Write This section describes how to monitor the amount of time it takes for each process to read from or write to any file. This is useful if you wish to determine what files are slow to load on a given system. iotime.stp iotime.stp tracks each time a system call opens, closes, reads from, and writes to a file. For each file any system call accesses, iotime.stp counts the number of microseconds it takes for any reads or writes to finish and tracks the amount of data (in bytes) read from or written to the file. iotime.stp also uses the local variable USDcount to track the amount of data (in bytes) that any system call attempts to read or write. Note that USDreturn (as used in disktop.stp from Section 4.2.1, "Summarizing Disk Read/Write Traffic" ) stores the actual amount of data read/written. USDcount can only be used on probes that track data reads or writes (for example syscall.read and syscall.write ). Example 4.6. iotime.stp Sample Output Example 4.6, "iotime.stp Sample Output" prints out the following data: A timestamp, in microseconds. Process ID and process name. An access or iotime flag. The file accessed. If a process was able to read or write any data, a pair of access and iotime lines should appear together. The access line's timestamp refers to the time that a given process started accessing a file; at the end of the line, it will show the amount of data read/written (in bytes). The iotime line will show the amount of time (in microseconds) that the process took in order to perform the read or write. If an access line is not followed by an iotime line, it simply means that the process did not read or write any data.

Commands:
global start global entry_io global fd_io global time_io function timestamp:long() { return gettimeofday_us() - start } function proc:string() { return sprintf("%d (%s)", pid(), execname()) } probe begin { start = gettimeofday_us() } global filenames global filehandles global fileread global filewrite probe syscall.open { filenames[pid()] = user_string(USDfilename) } probe syscall.open.return { if (USDreturn != -1) { filehandles[pid(), USDreturn] = filenames[pid()] fileread[pid(), USDreturn] = 0 filewrite[pid(), USDreturn] = 0 } else { printf("%d %s access %s fail\n", timestamp(), proc(), filenames[pid()]) } delete filenames[pid()] } probe syscall.read { if (USDcount > 0) { fileread[pid(), USDfd] += USDcount } t = gettimeofday_us(); p = pid() entry_io[p] = t fd_io[p] = USDfd } probe syscall.read.return { t = gettimeofday_us(); p = pid() fd = fd_io[p] time_io[p,fd] <<< t - entry_io[p] } probe syscall.write { if (USDcount > 0) { filewrite[pid(), USDfd] += USDcount } t = gettimeofday_us(); p = pid() entry_io[p] = t fd_io[p] = USDfd } probe syscall.write.return { t = gettimeofday_us(); p = pid() fd = fd_io[p] time_io[p,fd] <<< t - entry_io[p] } probe syscall.close { if (filehandles[pid(), USDfd] != "") { printf("%d %s access %s read: %d write: %d\n", timestamp(), proc(), filehandles[pid(), USDfd], fileread[pid(), USDfd], filewrite[pid(), USDfd]) if (@count(time_io[pid(), USDfd])) printf("%d %s iotime %s time: %d\n", timestamp(), proc(), filehandles[pid(), USDfd], @sum(time_io[pid(), USDfd])) } delete fileread[pid(), USDfd] delete filewrite[pid(), USDfd] delete filehandles[pid(), USDfd] delete fd_io[pid()] delete entry_io[pid()] delete time_io[pid(),USDfd] }
[...] 825946 3364 (NetworkManager) access /sys/class/net/eth0/carrier read: 8190 write: 0 825955 3364 (NetworkManager) iotime /sys/class/net/eth0/carrier time: 9 [...] 117061 2460 (pcscd) access /dev/bus/usb/003/001 read: 43 write: 0 117065 2460 (pcscd) iotime /dev/bus/usb/003/001 time: 7 [...] 3973737 2886 (sendmail) access /proc/loadavg read: 4096 write: 0 3973744 2886 (sendmail) iotime /proc/loadavg time: 11 [...]