lib/worker: Fix typo in scheduling reaping for jobs

Not quite sure what I was thinking there, but mea culpa for sure.

Signed-off-by: Andreas Ericsson <ae@op5.se>
core: Fix typo in worker handling code

When a helper is terminated by a signal, we must use WTERMSIG to
find out which signal it was. Previously we used WSTOPSIG, which
is really only useful if the child received SIGSTOP and is as
such nowhere near as useful.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: Reduce logging when reaping dormant jobs

We were quite spammy earlier. Since logmessages go through the socket
we don't want to be quite so spammy.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: Revamp reap timing somewhat for stale jobs

It's easy to artificially create a scenario where our immediate child
goes into uninterruptible sleep with some simple but clearly bananas
programming. Our immediate child, hereafter called "helper", must issue
the vfork() system call and its child may spawn a ton of offspring of
its own if it so wishes, and then go into sleep. If the child's
offspring (the helper's grandchildren) exit, they will become zombies
since the child doesn't wait() for them. The helper on the other hand
is stuck in uninterruptible sleep until its child has called either
exit() or execve(). If it instead calls sleep() the helper can get
stuck in uninterruptible sleep (ps state 'D') for quite some time.

The first time we attempt to reap such a job, the child and all the
childs' children will get the SIGKILL signal and, (unless they too
are stuck in uninterruptible sleep), exit promptly, but not get
wait()ed for, turning them into zombies. The helper on the other hand
will either get its signal queued for when it wakes up (on Linux) or
get its signal discarded (possibly other unices).

The helper will wake up when its immediate child dies, so apparently
that counts as "calling exit", but it will normally not have time to
wake up and smell the signal before the CPU is handed back to the
signalling process (ie, our worker). In such cases the helper will
be ready for reaping very, very soon, but not immediately. When that's
the case, we schedule a 0.25 second sleep before the first attempt at
reaping the check, and then try again every 5 seconds until we've
managed it.

In case the signal is lost instead of queued, that means we'll at
least have killed the helper 0.25 seconds after we get the code back,
and the worst case scenario is that the signal doesn't register until
after we go past the state where we check if the child has exited. In
such cases, we'll leave zombies hanging around for 5 seconds.

The scenario really should be very rare though, and especially on linux
where signals are delivered atomically.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: Add gcc printf attribute to wlog() function

It helps to get the warning if you miss an argument or some such.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: check for completion before we send SIGKILL

Sometimes helpers complete their work just before we attempt to reap
them, in which case the SIGKILL goes to a zombie and we're missing out
on the last chance to reap the actual result of the job instead.

This patch corrects that by forcing one last attempt at properly
reaping the child in case we're about to kill it.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: Try reaping processes when either I/O channel shuts down

It makes sense to do this, since some systems (notably Linux 3.8.0-25
from the Ubuntu repos with some wildly inappropriate sysctl settings)
fail to notify the parent process of stdout and stderr closing for
the child in case they've either had no output or use _exit() instead
of the more appropriate exit().

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: Catch SIGCHLD and schedule a "general" reaping when we do

This means we need a quick way to access the job we just reaped, so
we also add and maintain a fanout table for that precise purpose.

Since we don't expect to run more than a few hundred jobs at the same
time (at worst, that is), we use a relatively small one of 4096 entries
which seems to suffice just fine for our needs.

This appears to eliminate the "checks sometimes reach timeout without
being reaped" more or less completely, unless those checks actually
*do* reach their timeout.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: Unify job-state logging somewhat

We still only log exceptional states, but now we do it so that grep'ing
becomes a bit easier and the job id lines up properly with the pid
when viewing lists of the error messages.

Signed-off-by: Andreas Ericsson <ae@op5.se>