Fix EC2 instance type for a volume backed instance.

EC2 reports 'instance-store' type for a volume backed instance
which was booted with short root device name (no '/dev/' prefix).

This fix make instance type detection independent of addition
'dev' prefix to the instance root device name.

Change-Id: I6c2be09eaa91341e401bbc128b3efc79c60e345e
Closes-Bug: #1324400
Fix for volume detach error when use NFS as the cinder backend

For NFS volumes, every time you detach a volume, nova tries to umount
the device path. If the device path is busy (or in use), it should
output a message to log and continue.
In current code, if the device is busy, it cannot catch the ‘device is busy’
message returned by umount, because it looking for the ‘target is busy’.
 Therefore, current code skip the ‘if’ statement and run the ‘else’ and
raise the exception.
Fix: Add ‘device is busy’ to if statement.

Add a mock test to check the behaviour of the
virt.libvirt.volume.LibvirtNFSVolumeDriver.disconnect_volume
when it has umount errors.

Closes-Bug: #1340552

Change-Id: Iac946c37064c5f5bf5a102305de40d21d16846c1
Cells: Update set_admin_password for objects

Cells code was calling the compute/api set_admin_password method with an
old-world style instance and failing on the task_state attribute access.
This adds set_admin_password to the list of methods handled via the
cells RPC redirect method which is objects compatible.

Change-Id: I12e212260ed62dc9380202de31fda928a9498dcb
Closes-Bug: #1335315
Process power state syncs asynchronously

The instance locking introduced in
cc5388bbe81aba635fb757e202d860aeed98f3e8 keeps the power state sane
between stop and the periodic task power sync.  However, locking on
an instance in the periodic task thread can potentially lock
that thread for a long time.

Fix this condition by executing power syncs in a background GreenPool.
Limit the syncs to 1 at a time per instance.

Closes-Bug: 1363231

Change-Id: Idbc00ddfcb7a6816433863ed1686ad8f7477d99b
Remove snapshot_id from _volume_snapshot_create()

There is an argument "snapshot_id" in _volume_snapshot_create(), but
the argument is not used in _volume_snapshot_create().
This patch removes it for cleanup.

Change-Id: Ibaad290e149a863c749c74ace3ba702870b8dd1b
Allow forceDelete to delete running instances

Currently attempting to force delete a running instance
fails while a normal delete can succeed. This changes the
behaviour of force_delete so any instance can be deleted
and not just ones which have already been soft deleted.

Also catch InstanceIsLocked exceptions at the API layer
and now also don't need to catch InstanceInvalidState
exceptions since they are no longer raised

DocImpact: forceDelete will now delete any instance and not
  just ones which have already been soft deleted

Change-Id: I06a476701ccb47d8f7fd6e43c394ccb3fac2b3aa
Closes-Bug: 1351001
Fixes a Hyper-V list_instances localization issue

The Hyper-V WMI MSVM_ComputerSystem class Caption property can
have different values on various locales. This patch uses a
different query to avoid any localization issue.

Co-Authored-By: Robert Tingirica <rtingirica@cloudbasesolutions.com>
Change-Id: I15103b8edb3ff8934c835826ed00e08593cdeacc
Closes-Bug: #1296478
xenapi: send chunk terminator on subprocess exc

During chunk upload if a subprocess exception is thrown, a malformed
chunk upload occurs.

This patch adds logging for subprocess exceptions and sends the
terminator even in the event of a subprocess exception. The incomplete
file upload will then be cleaned up by the client.

Change-Id: I559f201430bcd4c36d669c7772a10bfe3cd54d41
Closes-Bug: 1365637
libvirt: improve unit test time

There were unit tests that were taking more than 4 seconds.

nova.tests.virt.libvirt.test_driver.LibvirtConnTestCase.test_clean_shutdown_failure    5.091
nova.tests.virt.libvirt.test_driver.LibvirtConnTestCase.test_clean_shutdown_with_retry 4.093

TrivialFix

Change-Id: I75cdb74dca69a1e31d6b974148c9590328c362e1
Closes-bug: #1366549
Fix config generator to use keystonemiddleware

The auth_token middleware was moved from keystoneclient.middleware
to keystonemiddleware. The config generator is configured to point
to the new location.

Closes-Bug: #1366594
Change-Id: I5543973b2c7ac06c048686c24354ecf6eb251027
Ironic:  Fix direct use of flavor and instance module objects

This replaces all uses of nova.objects.flavor_obj.Flavor* with
nova.objects.Flavor* and nova.objects.instance_obj.Instance* with
nova.objects.Instance*

This work was done throughout Nova. An example of this is
commit 3c1085f528596089a638ac3bb44a9ed3e201e62f.

Change-Id: I8f2dbb408acbdf8124bc141925de03ce6f12bd91
VMware: add tests for image fetch/cache functions

Adds unit tests for the vmops image fetch/prepare/cache functions.
No existing tests are modified.

partial blueprint vmware-spawn-refactor

Change-Id: I913c7f5ecfec586ff79e2bc7ab1969a8824476b5
Ironic: don't canonicalize extra_specs data

Don't canonicalize the cpu_arch extra_specs field. This is important
because the scheduler filters which use extra specs don't canonicalize
things either and as such you'll have mismatched fields causing
instances not to get scheduled.

This was recently changed as part of the Ironic -> Nova driver patch
series. We still want to canonicalize supported_instances though...

Change-Id: I9342213b5433113816142b1f737119065e9f077f
Closes-bug: #1366859
Add InstancePCIRequests object

Part of blueprint pci-passthrough-sriov

Co-Authored-By: Dan Smith <dansmith@redhat.com>
Co-Authored-By: Baodong (Robert) Li <baoli@cisco.com>
Change-Id: I125e4afdf98a7073abcb8117bbb7a3e3fe2f9830
Remove exclude coverage regex from coverage job

This commit removes the regex to exclude the coverage extension from
the tox coverage job. The regex was needed for the unit tests on the
coverage extension, (which would mock out coverage) however that
extension has since been removed from nova. So we no longer need to
exclude tests that don't exist anymore.

Change-Id: I45685b1113ecc78eda92f13755ff2274ff517829
Handle exception better in v2 attach_interface

Add more information when exception occurs in attach interface
at v2 API layer.

Change-Id: I7402cfc28d4ee7989c7eb054556df57918bbaeee
Fix 'os-start/os-stop' server actions for V2.1 API

'os-start/os-stop' server actions were missed for V2.1 API.
This patch converts 'os-start/os-stop' server action for V2.1 API

The differences between v2 and v3 are described on the wiki page
https://wiki.openstack.org/wiki/NovaAPIv2tov3.

Closes-Bug: #1367575

Change-Id: I8b0f27fb639034d368bcc1fc51d20daf9c2cdeb5
document why we disable usb_tablet in code

This provides a code level explanation for why we disregard the user
option for use_usb_tablet when the spice agent is enabled. It came up
in a bug report, and having the comment close to the code helps carry
the information forward.

Change-Id: I7945e921b6283519d91e234584d99d9e82218e6d
Closes-Bug: #1356633
libvirt: set pae for Xen PVM and HVM

The Physical Address Extension (PAE) enables paging to produce physical
addresses with more that 32 bit (2^32 bytes, 4 GBytes), namely it
extends the physical address space to 64 GBytes (2^36 bytes).

When spawning a 32 bit image in Xen PVM or HVM, PAE should be enabled in the
hypervisor so as to ensure that it is able to run properly. According to
the Xen documentation it can be enabled safely: "In general you should
leave this enabled and allow the guest Operating System to choose whether
or not to use PAE. " [1]

[1] http://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html#pae_boolean

Closes-Bug: #1100697
Change-Id: If8d8ebc38efd969d96a1953446d72e72691734b3
Stop stack tracing when trying to auto-stop a stopped instance

Commit cc5388bbe81aba635fb757e202d860aeed98f3e8 added locks to
stop_instance and the _sync_power_states periodic task to try and fix a
race between stopping the instance via the API where the task_state is
set to powering-off, and the periodic task seeing the instance
power_state as shutdown in _sync_instance_power_state and calling the
stop API again, at which point the task_state is already None from the
first stop API call and we get an UnexpectedTaskStateError.

The handle_lifecycle_event method is getting callbacks from the libvirt
driver on state changes on the VM and calling the
_sync_instance_power_state method which may try to stop the instance
asynchronously, and lead to UnexpectedTaskStateError if the instance is
already stopped by the time it gets the lock and the task_state has
changed.

Attempting to lock in handle_lifecycle_event just moves the race around
so this change adds logic to stop_instance such that if the instance
says it's active but the virt driver says it's not running, then we add
None to the expected_task_state so we don't stacktrace on
instance.save().

An alternative and/or additional change to this would be doing a call
rather than a cast when _sync_instance_power_state calls the stop API
but in some previous testing it doesn't appear to make a significant
difference in the race found when we hit the stop_instance method.

Adds a bunch of debug logging since this code is inherently racey and
is needed when looking at failures around these operations.

Closes-Bug: #1339235
Closes-Bug: #1266611
Related-Bug: #1320628

Change-Id: Ib495a5ab15de88051c5fa7abfb58a5445691dcad