Improve debug statements in death detection

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Niklas Hambuechen <niklash@google.com>
Report back death status after check

Make cleanupIfDead report the death status of the job,
so that a caller can decide to retry.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Niklas Hambuechen <niklash@google.com>
Add constants for retries of death detection

Add constants describing how often, and at which intervals,
to retry death detection of a job after sending it a SIGKILL.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Niklas Hambuechen <niklash@google.com>
Add orM from monad-loops

...a monadic short-circuit version of 'or'.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Niklas Hambuechen <niklash@google.com>
Retry death detection after killing

We cannot avoid the race on death detection after
forcefully killing a job: the only guarantee the
operating system gives us is that the process will
die eventually. However, we can improve the chance
of being able to successfully clean up a job by
retrying death detection. Do this.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Niklas Hambuechen <niklash@google.com>
Utils.Monad: Add unfoldrM and unfoldrM'

Taken from monad-loops, with a variable rename to avoid name shadowing.

Signed-off-by: Niklas Hambuechen <niklash@google.com>
TestCommon: Add listOfUniqueBy

This is a Gen for creating elements that are distinct by a certain property,
(e.g. for generating a list of jobs, all of which have different job IDs).

Signed-off-by: Niklas Hambuechen <niklash@google.com>
Test.JQScheduler: Fix queue jobs having same job IDs

The `Arbitrary Queue` instance so far would simply generated lists
of jobs, for which it was possible that job IDs are repeated
(inside individual lists or accross all lists).

While this hasn't created any problem so far, real-world Queues
cannot have repeated job IDs inside, and the Gen should reflect that.

Signed-off-by: Niklas Hambuechen <niklash@google.com>