Allow useage of server IDs as well as names.

RAX have some hidden images useful for build a xenserver host.
Ensure nodepool can use these by referring to them by image UUID.

Change-Id: Idfefc60e762740f4cffa437933007942a0920970
Make jenkins get info task synchronous

This is called within the main loop which means if there are a lot
of jenkins tasks pending, the main loop waits until they are all
finished.  Instead, make this synchronous.  It should be lightweight
so the additional load on jenkins should be negligible.

Also increase the sleep in the main loop to 10 seconds to mitigate
this somewhat.

Change-Id: I16e27fbd9bfef0617e35df08f4fd17bc2ead67b0
Merge "Allow useage of server IDs as well as names."
Fix typo in allocation

This was causing the target distribution for all images to be
calculated according to the current values of the last image
through the previous loop.

Change-Id: I3d5190c78849b77933e18c5cf6a9c7443945b6cd
Retry ssh connections on auth failure.

Some cloud instance types (Fedora for example) create
the ssh user after sshd comes online. This allows
our ssh connection retry loop to handle this scenario
gracefully.

Change-Id: Ie345dea50fc2983112cd2e72826a708583d2712a
Make nodepool more robust to offline clouds.

When a cloud is offline we cannot query it's flavors or extensions,
and without those we cannot use a provider manager. For these
attributes making the properties that lazy-initialize will fix the
problem (we may make multiple queries, but it is idempotent so
locking is not needed).

Callers that trigger flavor or extension lookups have to be able to
cope with a failure propogating up - I've manually found all the
places I think.

The catchall in _getFlavors would mask the problem and lead to
the manager being incorrectly initialized, so I have removed that.

Startup will no longer trigger cloud connections in the main thread,
it will all be deferred to worker threads such as ImageUpdate,
periodic check etc.

Additionally I've added some belts-and-braces catches to the two
key methods - launchImage and updateImage which while they don't
directly interact with a provider manager do access the provider
definition, which I think can lead to occasional skew between the
DB and the configuration - I'm not /sure/ they are needed, but
I'd rather be safe.

Change-Id: I7e8e16d5d4266c9424e4c27ebcc36ed7738bc86f
Fixes-Bug: #1281319
Merge "Make nodepool more robust to offline clouds."
Add fedora support

The previous loop iterating usernames doesn't work if another user is
added as we move onto the second username once ssh comes up and root
fails to run a command and then hit a timeout on the second user and
never attempt the 3rd.

Instead take an approach there we continue trying root until it either
works or ssh succeeds but a command can't be run. At this stage we can
try any other users that may be configured on the image, with a short
timeout (as we know ssh has come up, if it hadn't ssh'ing as root would
have timed out).

Change-Id: Id05aa186e8496d19f41d9c723260e2151deb45c9
Check server status in batch

And cache the results for 5 seconds.  This should remove a huge
amount of GETs for individual server status, instead replacing
them with a single request that fetches the status of all servers.
All individual build or delete threads will use the cached result
from the most recent server list, and if it is out of date, they
will trigger an update.

The main benefit of using the individual requests was that they
also provided progress information.  Since that just goes into
the logs and no one cares, we can certainly do without it.

Also includes a minor constant replacement.

Change-Id: I995c3f39e5c3cddc6f1b2ce6b91bcd178ef2fbb0
Perform all deletes in threads

Have all node deletions (like node launches) handled by a thread,
including ones started by the periodic cleanup.  This will make
the system more responsive when providers are unable to delete
nodes quickly, as well as when large numbers of nodes are deleted
by an operator, or when the system is restarted while many node
deletions are in progress.

Additionally, make the 'nodepool delete' command merely update the
db with the expectation that the next run of the cleanup cron will
spawn deletes.  Add a '--now' option so that an operator may still
delete nodes synchronously, for instance when the daemon is not
running.

Change-Id: I20ce2873172cb1906e7c5832ed2100e23f86e74e
Delete all building nodes on daemon start

Nodepool can not (currently) resume building a node if the daemon
is interrupted while that is happening.  At least have it clean
up nodes that are in the 'building' state when it starts.

Change-Id: I66124c598b01919d3fd8b6158c482d65508c6dae
Use the task manager to get extensions and flavors

Since these calls can now come from any thread, these should now
be run through the task manager to serialize them and make sure
that we don't run into novaclient thread-safety issues.

Change-Id: I46ab44b93d56ad1ce289bf837511b9373d3284ee
Node deletion related fixes

* Less verbose logging (these messages show up a lot).
* Handle the case where a db record disappears during cleanup but
  before the individual node cleanup has started.
* Fix a missed API cleanup where deleteNode was still being
  called with a node object instead of the id.

Change-Id: I2025ff19a51cfacff64dd8345eaf120bf3473ac2
Coerce all ids from novaclient to str

Nova might return int or str ids for various objects.  Our db
stores them all as strings (since that supports the superset).
Since we convert all novaclient info into simply list/dict
datastructures anyway, convert all the ids to str at the same
time to make comparisons elsewhere easier.

Change-Id: Ic90c07ec906865e975decee190c2e5a27ef7ef6d
Remove unhelpful log message

Since we're caching lists, we would actually expect a server not
to appear within the first two iterations of the loop, so remove
a log message that says it isn't there (which now shows up quite
often).

Change-Id: Ifbede6a141809e9fa40b910de2aabbd44f252fe5
Fix more str!=int bugs

When we return ids after creating servers or images, coerce them
to strings so they compare correctly when we wait for them.

Change-Id: I6d4575f9a392b6028bcec4ad57299b7f467cb764
Preserve HOLD state when job starts.

Previously when a job started on a node nodepool always changed that
node to state USED. Now preserve the old state value if it was
previously HOLD.

Change-Id: Ia4f736ae20b8b24ec079aa024fad404019725bcb
Keep current and previous snapshot images

The previous logic around how to keep images was not accomplishing
anything particularly useful.

Instead:
  * Delete images that are not configured or have no corresponding
    base image.
  * Keep the current and previous READY images.
  * Otherwise, delete any images that have been in their current
    state for more than 8 hours.

Also, correct the image-update command which no longer needs to
join a thread.  Also, fix up some poorly exercised parts of the
fake provider.

Change-Id: Iba921f26d971e56692b9104f9d7c531d955d17b4
Merge "Preserve HOLD state when job starts."
Merge "Retry ssh connections on auth failure."