drothlis/stb-tester — pull request #81 by asimihsan (2014-01-14 12:43:44+00:00)
Title: extra/runner/server - fix to stop zombie bash processes from building up

This commit fixes an issue with the stb-tester runner server that allows
zombie bash processes to build up over time. This is important because
zombie processes consume PIDs, which are a finite resource because they
are 16-bit unsigned shorts on Linux OSs. On a production system I have
seen eight zombie processes under `server`, which isn't terrible but
inevitably would get worse over time.

When a user edits either the notes field or the failure-reason field the
stb-tester runner server uses `subprocess.Popen` calls to start
instances of `extra/runner/report`, a Bash script, in order to
regenerate several HTML files. However, it never calls `wait()` on
the subprocess, and hence the subprocesses are guaranteed to become
zombie processes [1], indeed by definition.

A zombie process is a process that has completed execution but still has
an entry in the process table because it is waiting for its parent
process to read its exit status. Of course this being a web server you
do not want to synchronously execute the subprocess, or else you will
block waiting for it to complete, and for production systems perhaps
running on very slow backing stores large reports may take on the order
of minutes to generate.

Hence the correct solution is to use `threading`/`multiprocessing` to
start a new thread/process that itself will run the subprocess and
`wait()` on it. In this commit I've chosen to use `threading` but you
could use `multiprocessing` as well.

Moreover I've used decorators and `functools.wraps` to preserve the
`__name__`, `__doc__`, and `__module__` of the decorated function [2],
but this is just a nicety and could be removed for brevity if you wish.

[1] http://en.wikipedia.org/wiki/Zombie_process

[2] http://docs.python.org/2/library/functools.html#functools.wraps