Improve debug statements in death detection

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Niklas Hambuechen <niklash@google.com>
Report back death status after check

Make cleanupIfDead report the death status of the job,
so that a caller can decide to retry.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Niklas Hambuechen <niklash@google.com>
Add constants for retries of death detection

Add constants describing how often, and at which intervals,
to retry death detection of a job after sending it a SIGKILL.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Niklas Hambuechen <niklash@google.com>
Add orM from monad-loops

...a monadic short-circuit version of 'or'.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Niklas Hambuechen <niklash@google.com>
Retry death detection after killing

We cannot avoid the race on death detection after
forcefully killing a job: the only guarantee the
operating system gives us is that the process will
die eventually. However, we can improve the chance
of being able to successfully clean up a job by
retrying death detection. Do this.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Niklas Hambuechen <niklash@google.com>
Utils.Monad: Add unfoldrM and unfoldrM'

Taken from monad-loops, with a variable rename to avoid name shadowing.

Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
TestCommon: Add listOfUniqueBy

This is a Gen for creating elements that are distinct by a certain property,
(e.g. for generating a list of jobs, all of which have different job IDs).

Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Test.JQScheduler: Fix queue jobs having same job IDs

The `Arbitrary Queue` instance so far would simply generated lists
of jobs, for which it was possible that job IDs are repeated
(inside individual lists or accross all lists).

While this hasn't created any problem so far, real-world Queues
cannot have repeated job IDs inside, and the Gen should reflect that.

Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Add nodes field in class Disk

Add a field called "nodes" in the Disk class. This field will be a list
of Ganeti node UUIDs from which this disk is accessible.
Populate disk nodes when creating disks

When creating a Disk using the GenerateDiskTemplate function, fill the
nodes list depending on the disk template. There are three cases:

1) Internally mirrored disks (DRBD):
   These disks are accessible from two nodes, so the nodes list will
   include these. Their children (data, meta) are also accessible from
   two nodes, therefore they will inherit the nodes of the parent.
2) Externally mirrored disks (Blockdev, Ext, Gluster, RBD, Shared File):
   These disks should be accessible from any node in the cluster,
   therefore the nodes list will be empty.
3) Single-node disks (Plain, File):
   These disks are accessible from one node only, therefore the nodes
   list will consist only of the primary instance node.
Maintain consistency of nodes list

There are three scenarios in which the nodes list can change after the
disk's creation; when moving; replacing or recreating disks. For each
scenario, update the nodes list accordingly.
Update cfgupgrade tool

Create an upgrade function that adds a nodes list to each disk that is
attached to an instance. Also, add a downgrade function that removes the
nodes list and brings the config to the default state for 2.12.
Update various tests regarding disk creation

Update the tests that deal with disk creation and check if the nodes
list is populated correctly when creating a disk.