Adding runner monitoring for CI mode

In CI mode, if one of the runners aborted (say, due to SIGSEGV), the app
would sit and hang indefinitely. This monitors the running process and
quits if all of the runners have completed.