Check the return value of query in ConnectToInstanceConsole

When running gnt-instance console for an instance name that doesn't
exist, I got the following raw error:

...
    cl.QueryInstances([instance_name], ["console", "oper_state"], False)
ValueError: need more than 0 values to unpack

In this case we should raise the OpPrereqError to notify that the
instance which having that name doesn't exist.
This patch adds a check for the return value of the QueryInstances and
raise the OpPrereqError when the result of the query is a blank list.

Signed-off-by: Yuto KAWAMURA(kawamuray) <kawamuray.dadada@gmail.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Improve debug statements in death detection

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Niklas Hambuechen <niklash@google.com>
Report back death status after check

Make cleanupIfDead report the death status of the job,
so that a caller can decide to retry.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Niklas Hambuechen <niklash@google.com>
Add constants for retries of death detection

Add constants describing how often, and at which intervals,
to retry death detection of a job after sending it a SIGKILL.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Niklas Hambuechen <niklash@google.com>
Add orM from monad-loops

...a monadic short-circuit version of 'or'.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Niklas Hambuechen <niklash@google.com>
Retry death detection after killing

We cannot avoid the race on death detection after
forcefully killing a job: the only guarantee the
operating system gives us is that the process will
die eventually. However, we can improve the chance
of being able to successfully clean up a job by
retrying death detection. Do this.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Niklas Hambuechen <niklash@google.com>
Fix FIXME comment

As Ganeti supports jump-upgrades, we cannot anywhere in the
2.x series assume that we have not just been upgraded form
a 2.10 or earlier system. Hence the bootstrap code needs to
remain. Change the comment accordingly.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Utils.Monad: Add unfoldrM and unfoldrM'

Taken from monad-loops, with a variable rename to avoid name shadowing.

Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
TestCommon: Add listOfUniqueBy

This is a Gen for creating elements that are distinct by a certain property,
(e.g. for generating a list of jobs, all of which have different job IDs).

Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Test.JQScheduler: Fix queue jobs having same job IDs

The `Arbitrary Queue` instance so far would simply generated lists
of jobs, for which it was possible that job IDs are repeated
(inside individual lists or accross all lists).

While this hasn't created any problem so far, real-world Queues
cannot have repeated job IDs inside, and the Gen should reflect that.

Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Read if IPv6 is used directly in SshRunner

Moved check if the Primary Network is running on IPv6 into the SshRunner.
Until now SshRunner was only initialized correctly in the bootstrap.py
and not in the other 5 places.

Fixing issue #892

Signed-off-by: Neal Oakey <neal.oakey@googlemail.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Add withDefaultOnIOError

This is a specialized version of try with a constant
handler. The main benefit is the type annotation forcing
the free type variable in 'try' to be IOError.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Look up RPC port

...instead of hard-codedly using the default port.
Ganeti's noded honors /etc/services, so also look
up the port there when trying to contact the daemon.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Objects: Add filterRuleOrder / Ord instance for FilterRule

This is necessary to process filters in this order, as required
by the filtering spec.

Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Query.Filter: Add `evaluateFilterJSON`

We will need it to evaluate filters on OpCodes for job filtering.

Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
JSON: Add `nestedAccessByKey` and `nestedAccessByKeyDotted`

We will need them for job filters, where we want to access OpCodes
as a JSON structure.

Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Implement job filtering

This implements the operational part of the design doc
  "Filtering of jobs for the Ganeti job queue"
(design-optables.rst).

It includes
- respecting filter rules when jobs are scheduled
- cancelling running jobs rejected by filters
- re-running the scheduler when filter rules are changed
- handling of the filter actions ACCEPT, CONTINUE, PAUSE, REJECT
  and RATE_LIMIT
- implementation of the "jobid", "opcode" and "reason" predicates

Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Unit tests for job filtering

This tests only the filter logic; the QA testing filter management + logic
comes later.

Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
qa_utils: Make AssertCommand return stdout/stderr as well

This is usefull if the run commands output something that's needed afterwards.

Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
qa_utils: Allow passing fail=None to AssertCommand

This is for cases where we don't care about the exit code.

Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>