draios/sysdig — pull request #113 by brendangregg (2014-04-13 21:03:04+00:00)
Title: fileslower tool for investigating latency outliers

This chisel allows quick analysis of file I/O latency outliers. For example, those latency greater than 1ms:

```
# sysdig -c fileslower 1
TIME PROCESS TYPE LAT(ms) FILE
2014-04-13 20:40:43.973 cksum read 2 /mnt/partial.0.0
2014-04-13 20:40:44.187 cksum read 1 /mnt/partial.0.0
2014-04-13 20:40:44.689 cksum read 2 /mnt/partial.0.0
2014-04-13 20:40:45.005 cksum read 2 /mnt/partial.0.0
2014-04-13 20:40:45.193 cksum read 1 /mnt/partial.0.0
[...]
```

Typical use will be checking for latency greater than 10s or 100s of ms. This is particularly useful for exonerating file system and storage devices, proving that file I/O (of the type traced) has not exceeded expected latencies.

A minimum latency of 0 will show all file I/O:

```
# sysdig -c fileslower 0
TIME PROCESS TYPE LAT(ms) FILE
2014-04-13 20:59:04.414 ls read 0 /lib/x86_64-linux-gnu/libselinux.so.1
2014-04-13 20:59:04.414 ls read 0 /lib/x86_64-linux-gnu/librt.so.1
2014-04-13 20:59:04.414 ls read 0 /lib/x86_64-linux-gnu/libacl.so.1
2014-04-13 20:59:04.414 ls read 0 /lib/x86_64-linux-gnu/libc.so.6
2014-04-13 20:59:04.414 ls read 0 /lib/x86_64-linux-gnu/libdl.so.2
2014-04-13 20:59:04.414 ls read 0 /lib/x86_64-linux-gnu/libpthread.so.0
2014-04-13 20:59:04.414 ls read 0 /lib/x86_64-linux-gnu/libattr.so.1
2014-04-13 20:59:04.415 ls read 0 /proc/filesystems
2014-04-13 20:59:04.415 ls read 0 /proc/filesystems
2014-04-13 20:59:04.418 date read 0 /lib/x86_64-linux-gnu/librt.so.1
2014-04-13 20:59:04.418 date read 0 /lib/x86_64-linux-gnu/libc.so.6
2014-04-13 20:59:04.418 date read 0 /lib/x86_64-linux-gnu/libpthread.so.0
2014-04-13 20:59:04.418 date read 0 /etc/localtime
2014-04-13 20:59:04.418 date read 0 /etc/localtime
```

The latency for these reads were all 0 milliseconds (rounded), due to caching in the file system. The tool can be easily modified to show microseconds or nanoseconds, but given the role -- analyzing latency outliers -- using milliseconds is most effective in this case.