TechEmpower/FrameworkBenchmarks — pull request #681 by achlipala (2013-12-29 14:39:37+00:00)
Title: Toolset: A start at capturing performance stats during benchmark executi...

I hope this code is along the lines of what you were looking for, to capture and save performance measurements during the execution of benchmarks.

I'm relatively sure that this change captures the one piece of information I was interested in, which is RAM usage at the end of each test. Some frameworks might do funky things with giving allocated memory back to the system, but probably 95% of the useful information is contained in RAM usage right after wrk is done.

Most of the code in this commit relies on the Python psutil library, but I'm doing something OS-specific for memory usage, since psutil doesn't seem to provide the right granularity of information. On Linux, I'm poking around /proc to get what some web research tells me is the best approximation of how much RAM processes actually take up. Since I only use Linux, there's placeholder code now for other OSes.

So, like I said, probably this code captures the one number I'm most interested in, but I've also tried to include the other measurements that you asked for. Each stats snapshot measures CPU, disk, and network usage, in some way that may or may not be what you had in mind. I also take a number of stats snapshots per test execution. Currently, the snapshots before the final one are set to be run at predetermined time intervals after the test started. This may not be the best idea, and in any case, the arbitrary intervals I've included here probably deserve tweaking.

You might also have some opinions about the way I've modified the JSON benchmark result output, but I think all the information is there now.

I'm looking forward to seeing memory numbers especially to compare frameworks, so I hope to see some code like this used in Round 9! Please let me know if I can help get it into suitable shape.