Unmount the NFS and GlusterFS shares on detach

When attaching an NFS or GlusterFS volume, nova mounts the share so it
can be reused by other instances but it is not unmounted when detaching.

This patch adds the disconnect_volume method to both drivers, which will
try to unmount the share. If a volume in that share is still being used
by another instance, the umount command will fail with the error 'target
is busy' and we will ignore it. Any other error while unmounting will be
properly logged. In any case the detach will fail because nova hasn't
been able to unmount the share.

Closes-Bug: #1285209
Change-Id: I79289524541a88ad359b3465719055cd402b4e3d
xenapi: fix spawn servers with ephemeral disks

The new resize ephemeral disk code uses vdis["ephemerals"] to pass the
list of ephemeral vdis to the resize disk code. This is shared between
spawn and resize. As spawn wants to resize the root disk, resize=True
during the spawn operation.

The code in _attach_disks correctly deals with empty ephemeral disks,
but the code in _resize_up_vdis did not, and failed due to
ephemeral_vdis being None.

This is a quick fix for the issue. We need some refactoring in this area
so there is, ideally, only one place that generates ephemeral disks.

Change-Id: If058c5c17c2ed76569fd98d4a3d8e5274721fbd0
Closes-Bug: #1289361
Fix debug message formatting in server_external_events

Hopefully this will be the last case of invalid debug log messages to
fix before turning on debug level log messages on during unit tests, so
we can gate on issues like this.

Change-Id: I8a2b64222deaf5320bcc2e5215b6bd46565a505f
Use debug level logging during unit tests

When unit tests fail we need to debug, so we should have debug level logging on.
We use fixtures FakeLogger during testing, to  enable debug logging pass
in log level as a parameter.
http://bazaar.launchpad.net/~python-fixtures/python-fixtures/trunk/view/head:/lib/fixtures/_fixtures/logger.py#L68

Change-Id: I70790aa39c0774726de71e5fa5751c45e7e34ffd
Use instance object instead of _instance_update()

Currently, if a user takes a snapshot of a VM, stops nova-cpu
before the snapshot operation is finished, and restart nova-cpu:
nova-cpu will crash with the error:
AttributeError: 'dict' object has no attribute 'task_state'
This happens because the _instance_update() method returns a dict
and not an Instance object.

This patch fixes the issue by not using _instance_update()
in the compute manager which is returning a dict and using
the instance object instead.

Change-Id: I3af0b09aeff7790b147a684f7c90af17d8ee7ae2
Closes-Bug: #1287945
Fix: Unshelving an instance uses original image

The driver uses the property image_ref to boot the
the instance.
This patch updates the image_ref property with the
shelved_image_id stored during the shelving.

Change-Id: I3bba0a230044613e07122a6d122597e5b8d43438
Closes-Bug: #1257594
Fix compute_node stats

The compute_node object has several issues with the stats field
that are cleaned up by this patch, including the following:

- obj_make_compatible() should check the stats property exists
  before attempting to delete it
- from_db_object() should not create the stats property if it
  is not in the database field (it is a nullable field)
- create() should convert the stats property to its database
  field format if it is present
- tests for create() and save() should check that the database
  format conversion is done

This method does not change any fields so there is no change to
the object's version.

Change-Id: I5c68994d618f0ecf9409c37321bc87a006c431ac
Task cleanup_running_deleted_instances can now use slave

Deployments that have a slave_connection configured can offload
read queries generated by this periodic task there. For
deployments with no slave, behavior remains the same.

Partially-implements: blueprint periodic-tasks-to-db-slave
Change-Id: Ie1e6cdfbefab096a2af62ef636786a5c8f659b13
Merge "Use instance object instead of _instance_update()"
Merge "Fix debug message formatting in server_external_events"
Merge "Use debug level logging during unit tests"
Merge "xenapi: fix spawn servers with ephemeral disks"
Delete ERROR+DELETING VMs during compute startup.

We should perhaps do this check during message bus reconnection as
well.. Anyhow, if a compute node is offline during a nova API call
to delete an instance, and the rabbit message is lost for some
reason (or alternatively if the delete method throws an error)
then the task state is not cleared and won't be cleared on compute
restart, leaving it wedged forever.

Change-Id: Ie0a47958eb0fb58307902437a95634d5f54f74f3
Fixes-bug: #1281324
Co-Authored-By: Steve Kowalik <steven@wedontsleep.org>
Fix get_download_hander() typo

Clearly, this was meant to be get_download_handler().

Refactor loading modules in GlanceImageService a little to avoid the
get_download_handler() going over the length-of-line limit.

Change-Id: If871413cbb86cd800be8517bb37932df890bf53d
Add py27local tox target

This adds a new tox target that uses sitepackages=False. This is
handy for people running unit tests on a devstack box, where the
site-packages oslo.config interferes.

Change-Id: I1101730f18550de0b391d12bcc5841cb671434a2
VMware: image cache aging

The current patch set deletes cached images that are no longer used.
Images that are unused will be marked for aging. These images
will be deleted after they have not been used for
'remove_unused_original_minimum_age_seconds'.

The third and final phase of blueprint vmware-image-cache-management.

NOTE: in order to ensure that there will not be race conditions with
multiple compute nodes, each compute node will have its own datastore
cache directory. This will be a combination of the configuration
variables my_ip and image_cache_subdirectory_name. If the aging is
disabled then the original base is used (ensures backwards
compatibility).

Locking is done when accessing timestamps to detect if an image has aged
in the cache.

Change-Id: Iec47c28a38761c187226c5eff3ab69da503437f6
Updated from global requirements

Change-Id: I0b91fc1bb5e059cd5d56c3cc3557ee2986d811e4
xenapi: Speed up tests by not waiting on conductor

Part of testing the xenapi agent code requires creating faults. This is
usually sent to the conductor to add to the database, but no conductor
is running during the tests. This results in waiting for a 15 second
timeout for every test that does this.

Disabling use of the conductor during these tests shaves about 45 seconds
off that is spent waiting on a reply that will never come.

Before:
    test_spawn_agent_upgrade_fails_silently  OK  15.20
    test_spawn_fails_agent_not_implemented   OK  15.18
    test_spawn_fails_with_agent_bad_return   OK  15.17

After:
    test_spawn_agent_upgrade_fails_silently  OK  0.16
    test_spawn_fails_agent_not_implemented   OK  0.18
    test_spawn_fails_with_agent_bad_return   OK  0.18

Change-Id: I31fedf6710aba3778eca694b45aabcffe4d09133
Remove the docker driver

The docker compute driver does not currently have CI running against
it.  In accordance with our plan to require this for all drivers, we
are now removing the driver from nova.

https://wiki.openstack.org/wiki/HypervisorSupportMatrix/DeprecationPlan

At this point, the driver should be set up in a stackforge repository
and have CI staged from there to prove its reliability before we could
consider bringing it back into Nova.

DocImpact
UpgradeImpact

Change-Id: I1c9bea2fdeebc4199c4f7d8fca4580a6fb7fed5b
Merge "Unmount the NFS and GlusterFS shares on detach"