Package: schniepel
SchniepelMeasures the resource utilization of a specific process over time.This program also measures the utilization / saturation of system-wide resources
making it straightforward to put the process-specific metrics into context.Built for Linux. Windows and Mac OS support might come.Highlights:High sampling rate: by default, Schniepel uses a sampling interval of 0.5 seconds
for making narrow spikes visible.Schniepel is built for monitoring a program subject to process ID changes. This
is useful for longevity experiments when the monitored process occasionaly
restarts (for instance as of fail-over scenarios).Schniepel can run unsupervised and infinitely long with predictable disk space
requirements (it applies an output file rotation and retention policy).Schniepel helps keeping data organized: time series data is written intoHDF5files, and
annotated with relevant metadata such as the program invocation time, system
hostname, and Schniepel software version.Schniepel comes with a data plotting tool (separate from the data acquisition
program).Schniepel values measurement correctness very highly. The core sampling loop does
little work besides the measurement itself: it writes each sample to a queue.
A separate process consumes this queue and persists the time series data to
disk, for later inspection. This keeps the sampling rate predictable upon disk
write latency spikes, or generally upon backpressure. This matters especially
in cloud environments where we sometimes see fsync latencies of multiple
seconds.MotivationThis was born out of a need for solid tooling. We started withpidstat from
sysstat, launched aspidstat -hud -p $PID 1 1. We found that it does not properly account for
multiple threads running in the same process, and that various issues in that
regard exist in this program across various versions (seehere,here, andhere).The programcpustatopen-sourced by
Uber has a delightful README about the general measurement methodology and
overall seems to be a great tool. However, it seems to be optimized for
interactive usage (whereas we were looking for a robust measurement program
which can be pointed at a process and then be left unattended for a significant
while) and there does not seem to be a decent approach towards persisting the
collected time series data on disk for later inspection (it seems to be able to
write a binary file when using-cpuprofilebut it is a little unclear what
this file contains and how to analyze the data).The programpsrecord(which effectively
wrapspsutil) has a similar
fundamental idea as Schniepel; it however does not have a clear separation of
concerns between persisting the data to disk, performing the measurement itself,
and plotting the data, making it too error-prone and not production-ready.UsageHints and tricksConvert an HDF5 file to a CSV fileI recommend de-serialize and re-serialize usingpandas. Example one-liner:python -c 'import sys; import pandas as pd; df = pd.read_hdf(sys.argv[1], key="schniepel_timeseries"); df.to_csv(sys.argv[2], index=False)' messer_20190718_213115.hdf5.0001 /tmp/hdf5-as-csv.csvNote that this significantly inflates the file size (e.g., from 50 MiB to 300
MiB).NotesSchniepel tries to not asymmetrically hide measurement uncertainty. For example,
you might see it measure a CPU utilization of a single-threaded process
slightly larger than 100 %. That's simply the measurement error. In other
tooling such assysstatit seems to be common practice toasymmetricallyhide measurement uncertainty by capping values when they are known to in
theory not exceed a certain threshold
(example).Must be run withrootprivileges.The value-1has a special meaning for some metrics
(NaN, which cannot be represented
properly in HDF5). Example: A disk write latency of-1 msmeans that no
write happened in the corresponding time interval.The highest meaningful sampling rate is limited by the kernel's timer and
bookkeeping system.Measurands (columns, and their units)The quantities intended to be measured.proc_cpu_idThe ID of the CPU that this process is currently running on.Momentary state at sampling time.proc_cpu_util_percent_totalThe CPU utilization of the process inpercent.Mean over the past sampling interval.If the inspected process is known to contain just a single thread then
this can still sometimes be larger than 100 % as of measurement errors. If the
process contains more than one thread then this can go far beyond 100 %.This is based on the sum of the time spent in user space and in kernel space.
For a more fine-grained picture the following two metrics are also available:proc_cpu_util_percent_user, andproc_cpu_util_percent_system.proc_num_threadsThe number of threads in the process.Momentary state at sampling time.proc_num_ip_sockets_openThe number of sockets currently being open. This includes IPv4 and IPv6 and does
not distinguish between TCP and UDP, and the connection state also does not
matter.Momentary state at sampling time.proc_num_fdsThe number of file descriptors currently opened by this process.Momentary state at sampling time.proc_disk_read_throughput_mibpsandproc_disk_write_throughput_mibpsThe disk I/O throughput of the inspected process, inMiB/s.Based on Linux'/proc/<pid>/iorcharandwchar. A highly relevantpiece of documentation(emphasis mine):The number of bytes which this task has caused to be read from storage. This
is simply the sum of bytes which this process passed to read() and pread().It includes things like tty IOand it is unaffected by whether or not actual
physical disk IO was required (the read might have been satisfied from
pagecache)Mean over the past sampling interval.proc_mem_rss_percentFraction of processresident set size(RSS) relative to machine's physical memory size inpercent.Momentary state at sampling time.proc_ctx_switch_rate_hzThe rate of (voluntary and
involuntary) context switches inHz.Mean over the past sampling interval.(list incomplete)Valuable referencesExternal references on the subject matter that I found useful during
development.About system performance measurement, and kernel time bookkeeping:http://www.brendangregg.com/usemethod.htmlhttps://www.vividcortex.com/blog/monitoring-and-observability-with-use-and-redhttps://github.com/uber-common/cpustat/blob/master/README.mdhttps://elinux.org/Kernel_Timer_Systemshttps://github.com/Leo-G/DevopsWiki/wiki/How-Linux-CPU-Usage-Time-and-Percentage-is-calculatedAbout disk I/O statistics:https://www.xaprb.com/blog/2010/01/09/how-linux-iostat-computes-its-results/https://www.kernel.org/doc/Documentation/iostats.txthttps://blog.serverfault.com/2010/07/06/777852755/(interpreting iostat output)https://unix.stackexchange.com/a/462732(What are merged writes?)https://stackoverflow.com/a/8512978(what is%utilin iostat?)https://coderwall.com/p/utc42q/understanding-iostathttps://www.percona.com/doc/percona-toolkit/LATEST/pt-diskstats.htmlOthers:https://serverfault.com/a/85481/121951(about system memory statistics)Musings about HDF5:https://cyrille.rossant.net/moving-away-hdf5/http://hdf-forum.184993.n3.nabble.com/File-corruption-and-hdf5-design-considerations-td4025305.htmlhttps://pytables-users.narkive.com/QH2WlyqN/corrupt-hdf5-fileshttps://www.hdfgroup.org/2015/05/whats-coming-in-the-hdf5-1-10-0-release/https://stackoverflow.com/q/35837243/145400