Fix instance can not be deleted after soft reboot

The reason is that:
1. soft reboot will wait for instance become to 'shutdown', and then start it
2. delete operation also wait for this, and then clean up the instance
3. if soft reboot found the instance become to 'shutdown' firstly, it will
start it immediately
4. then the delete operation will go to the _wait_for_destroy loop, and the
loop may be endless
5. when we delete the instance again, because the lock was hold by the delete
operation before, this one will wait the lock and don't implement actually.
So the domain id is checked during _wait_for_destroy loop, if it changed and
the instance is still running, we need to destroy it again.
If the domain is booted after _wait_for_destroy, it may result in
unfilter_instance failed because the nwfilter is in use, so doing the same
thing as above.

Fixes Bug #1111213

Change-Id: I98dc902dd86fa828f5821465c611953e08f9f637
Retry bw_usage_update() on innodb Deadlock

Adds a new decorator _retry_on_deadlock() to sqlalchemy api.  This patch
makes bw_usage_update() use it.

Fixes bug 1129622

Change-Id: I0293c62d2dd5ac036445bc639cabbd05ba016e83
Add an extension to show image size.

Adds image size data that is returned from Glance into the API response.
This becomes really useful for users when they take a snapshot and want
to track the storage used by it.  Since this information is already
being pulled by Nova we should display it.

DocImpact
Change-Id: I8a2d4001c21bcc54cac1a2992034dfc9fbe39d7e
Add snapshot methods to fakes.py

Modified Snapshots unit tests to use them
Removed logging from unit tests
The methods will also be used by the sample tests
Fixes bug 1130313

Change-Id: Id6c9c809952acefe302432bdfc345378730f0243
Fix instance directory path for lxc

Commit 12b8bd380e22a40e10e9d2758f5a0d594517de51 triggers a regression
when starting a STOPPED LXC instance. Instance referecne is needed to
correctly setup container rootfs path.

Change-Id: Id55b7f3a631e80a0da3f52c55784414105a61142
Fix pep8 issues with test_manager.py

Fixes E231 (missing whitespace) problems with
nova/tests/network/test_manager.py

Change-Id: I99d4c3bf7ed1a137fbfb26e9fe2f31d9104d5ae4
More gracefully handle TimeoutException in test

Sometimes although the test_killed_worker_recover test passes, the
cleanup of a forked child takes too long and a TimeoutException
occurs. This patch better handles the TimeoutException and
ensures the child process is killed.

Fixes bug #1128574

Change-Id: I39ea2fbb6c9cb95ad10b68bfa216686467951a93
ensure we run db tests in CI

CI had a set of environmental variables that refused to allow the
tests from failing gracefully. These seem to have gotten lost
along the way in refactoring.

Change-Id: Ib19b5d472c51ed72b8acf1571dd19f41474532c5
Refactor nwfilter parameters

* libvirt < 1.0.1 does not allow dynamically updating filter
  parameters inside an interface block of theinstance XML. To
  address this we move the nwfilter paramters from the instance
  XML to the per-instance base nwfilter.

Relates to bug #1124722

Change-Id: I50ec5dcffa8ff0232ae0aba0a08fce5151962238
Make add-fixed-ip update nwfilter wth in libvirt

Add inject_network_info to the libvirt driver  so that the
add-fixed-ip call will trigger a refresh of the nwfilters.

Change-Id: I00eccf34b8f92fb1e26a72e4bddb47dd39c4558f
Fixes: bug #1124722
Adding ability to specify the libvirt cache mode for disk devices

In past versions of Nova it was possible to explicitly configure
the cache mode of disks via the libvirt XML template. The current approach
makes this a derived setting of either “none” or “writethrough” based
on the support of O_DIRECT. Whilst this provides a good set of default
settings it removes the ability of the cloud provider to use other
modes such as “writeback” and “unsafe” which are valuable in certain
configurations.

This change allows the cache mode to be specified on a per-disk type
basis. If a disk type does not have a cache mode specified then the
default behaviour remains unchanged.

DocImpact

bug 1086386

Change-Id: I3d296fe0b4b9b976db02db90ad69fd299cd2096a
Removed duplicate spawn code in PowerVM driver

After introducing resize/migrate in PowerVM there was
some code that was duplicated after a rebase. The commit
removes the duplicate code. The functionality of
resize/migrate remains the same as introducted in
Change I72345626

Bug #1130680

Change-Id: I1fe3d8e1f2a19fd40a03c8c38d90c01711356b38
Improve I/O performance for periodic tasks

Unfortunately, periodic tasks to update local resource and
driver status come with performance penalties, especially on LVM
image backend type.

This is because low level implementation relies 'vgs' command to
query LVM info, which itself has a I/O performance price and
extravagant use of local_gb_xxx query make things even worse.

This commit centralize such query in one single call
(volume_group_info) and reduce invocation times in periodic tasks
accordingly.

Partially resolves LP #1130697

Change-Id: I165f96dd3e4fdad770b8a9643f5a038d353312ff
Add better status to baremetal deployments.

This patch introduces a few new baremetal states, which are used to
track the deploy process. Now, nova-baremetal-deploy-helper updates the
bm_nodes record directly when it begins and finishes deploying an image
to that node.

The next patch will add a LoopingCall inside driver.spawn() to wait for
the deploy to complete.

Also, since there can not be >1 active deployment per node, there
is no need to have a separate table for storing them. This patch drops
the table bm_deployments and adds the important information it contained
to bm_nodes. Since the previous behavior was to mark a deployment as
deleted once it completed, there is no need to copy any data from
bm_deployments prior to dropping the table -- assuming that no active
deployments are in process when the migration is run.

Since this is the first migration for the baremetal database, it also
adds a new test class, TestBaremetalMigrations, and refactors the
test_migrations.py file to allow for multiple test classes.

partially implements fix for bug 1096723

Change-Id: Iad30b462d49c88fc19babed43a2fb8540b1fad30
libvirt: fix volume walk of /dev/disk/by-path

If /dev/disk/by-path does not exist, then _get_iscsi_devices() will
raise an IndexError. Handle this case and return an empty list. This
fixes a test on OSX.

Fixes bug 1130841

Change-Id: Ibae2f1238feac9a10b8dc2b74886ea59edac0bcc
Make nova security groups more pluggable

This patch moves the nova security group code out of nova/compute/api.py
into nova/network/security_group. It also removes any query to the database
from security group api into the nova security group driver. This allows
security group drivers the ability to decouple themselves from storing
security group information in the nova_db.

Change-Id: Ib183515a0418203c8bcc88176e3a1498d7333300
Add a volume driver in Nova for Scality SOFS

Scality SOFS is a network filesystem mounted with FUSE, with most
options given in a configuration file. Given a mount point and a SOFS
configuration file as driver options, the Scality volume driver mounts
SOFS, and then creates, accesses and deletes volumes as regular
(sparse) files on SOFS.

Change-Id: I84bf268e5a2c5c33b8706830e8067914fae44aed
Implements: blueprint scality-volume-driver
Add Nova quantum security group proxy

Implements blueprint nova-quantum-security-group-proxy

Change-Id: I21d70f1dc8e61d6412d14e30ab2aa1a83a711de2
cfg should be imported from oslo.config

Change-Id: Idd9c3ededd2f63156eb78ed01fd38ddb3a20437f
Wait for baremetal deploy inside driver.spawn

Previously, baremetal driver.spawn returned as soon as the
machine power turned on, but before the user-image was deployed to the
hardware node, and long before the node was available on the network.
This meant the nova instance was marked as ACTIVE before provisioning
had actually finished. If the deploy failed and the baremetal node was
set to an ERROR state, the nova instance could still be left as ACTIVE
and the user was never informed of the error.

This patch introduces a LoopingCall to monitor the deployment status in
the baremetal database. As the deployment is performed by
nova-baremetal-deploy-helper, the database record is updated. Once the
deployment is complete, driver.spawn() sets the baremetal node status
and the nova instance status is also set properly. If an error occurs
during the deployment, an exception is raised within driver.spawn()
allowing nova to follow the normal cleanup and notify paths.

This also allows the baremetal PXE driver to delete cached image files
when a baremetal deployment fails.

Fixes bug 1088655.

Change-Id: I4feefd462fd956c9780995ec8b05b13e78278c8b