Fix instance can not be deleted after soft reboot

The reason is that:
1. soft reboot will wait for instance become to 'shutdown', and then start it
2. delete operation also wait for this, and then clean up the instance
3. if soft reboot found the instance become to 'shutdown' firstly, it will
start it immediately
4. then the delete operation will go to the _wait_for_destroy loop, and the
loop may be endless
5. when we delete the instance again, because the lock was hold by the delete
operation before, this one will wait the lock and don't implement actually.
So the domain id is checked during _wait_for_destroy loop, if it changed and
the instance is still running, we need to destroy it again.
If the domain is booted after _wait_for_destroy, it may result in
unfilter_instance failed because the nwfilter is in use, so doing the same
thing as above.

Fixes Bug #1111213

Change-Id: I98dc902dd86fa828f5821465c611953e08f9f637
Retry bw_usage_update() on innodb Deadlock

Adds a new decorator _retry_on_deadlock() to sqlalchemy api.  This patch
makes bw_usage_update() use it.

Fixes bug 1129622

Change-Id: I0293c62d2dd5ac036445bc639cabbd05ba016e83
Add an extension to show image size.

Adds image size data that is returned from Glance into the API response.
This becomes really useful for users when they take a snapshot and want
to track the storage used by it.  Since this information is already
being pulled by Nova we should display it.

DocImpact
Change-Id: I8a2d4001c21bcc54cac1a2992034dfc9fbe39d7e
Add snapshot methods to fakes.py

Modified Snapshots unit tests to use them
Removed logging from unit tests
The methods will also be used by the sample tests
Fixes bug 1130313

Change-Id: Id6c9c809952acefe302432bdfc345378730f0243
Fix instance directory path for lxc

Commit 12b8bd380e22a40e10e9d2758f5a0d594517de51 triggers a regression
when starting a STOPPED LXC instance. Instance referecne is needed to
correctly setup container rootfs path.

Change-Id: Id55b7f3a631e80a0da3f52c55784414105a61142
Fix pep8 issues with test_manager.py

Fixes E231 (missing whitespace) problems with
nova/tests/network/test_manager.py

Change-Id: I99d4c3bf7ed1a137fbfb26e9fe2f31d9104d5ae4
More gracefully handle TimeoutException in test

Sometimes although the test_killed_worker_recover test passes, the
cleanup of a forked child takes too long and a TimeoutException
occurs. This patch better handles the TimeoutException and
ensures the child process is killed.

Fixes bug #1128574

Change-Id: I39ea2fbb6c9cb95ad10b68bfa216686467951a93
ensure we run db tests in CI

CI had a set of environmental variables that refused to allow the
tests from failing gracefully. These seem to have gotten lost
along the way in refactoring.

Change-Id: Ib19b5d472c51ed72b8acf1571dd19f41474532c5
Refactor nwfilter parameters

* libvirt < 1.0.1 does not allow dynamically updating filter
  parameters inside an interface block of theinstance XML. To
  address this we move the nwfilter paramters from the instance
  XML to the per-instance base nwfilter.

Relates to bug #1124722

Change-Id: I50ec5dcffa8ff0232ae0aba0a08fce5151962238
Make add-fixed-ip update nwfilter wth in libvirt

Add inject_network_info to the libvirt driver  so that the
add-fixed-ip call will trigger a refresh of the nwfilters.

Change-Id: I00eccf34b8f92fb1e26a72e4bddb47dd39c4558f
Fixes: bug #1124722
Removed duplicate spawn code in PowerVM driver

After introducing resize/migrate in PowerVM there was
some code that was duplicated after a rebase. The commit
removes the duplicate code. The functionality of
resize/migrate remains the same as introducted in
Change I72345626

Bug #1130680

Change-Id: I1fe3d8e1f2a19fd40a03c8c38d90c01711356b38
Improve I/O performance for periodic tasks

Unfortunately, periodic tasks to update local resource and
driver status come with performance penalties, especially on LVM
image backend type.

This is because low level implementation relies 'vgs' command to
query LVM info, which itself has a I/O performance price and
extravagant use of local_gb_xxx query make things even worse.

This commit centralize such query in one single call
(volume_group_info) and reduce invocation times in periodic tasks
accordingly.

Partially resolves LP #1130697

Change-Id: I165f96dd3e4fdad770b8a9643f5a038d353312ff
libvirt: fix volume walk of /dev/disk/by-path

If /dev/disk/by-path does not exist, then _get_iscsi_devices() will
raise an IndexError. Handle this case and return an empty list. This
fixes a test on OSX.

Fixes bug 1130841

Change-Id: Ibae2f1238feac9a10b8dc2b74886ea59edac0bcc
Make nova security groups more pluggable

This patch moves the nova security group code out of nova/compute/api.py
into nova/network/security_group. It also removes any query to the database
from security group api into the nova security group driver. This allows
security group drivers the ability to decouple themselves from storing
security group information in the nova_db.

Change-Id: Ib183515a0418203c8bcc88176e3a1498d7333300
Add Nova quantum security group proxy

Implements blueprint nova-quantum-security-group-proxy

Change-Id: I21d70f1dc8e61d6412d14e30ab2aa1a83a711de2
Wait for baremetal deploy inside driver.spawn

Previously, baremetal driver.spawn returned as soon as the
machine power turned on, but before the user-image was deployed to the
hardware node, and long before the node was available on the network.
This meant the nova instance was marked as ACTIVE before provisioning
had actually finished. If the deploy failed and the baremetal node was
set to an ERROR state, the nova instance could still be left as ACTIVE
and the user was never informed of the error.

This patch introduces a LoopingCall to monitor the deployment status in
the baremetal database. As the deployment is performed by
nova-baremetal-deploy-helper, the database record is updated. Once the
deployment is complete, driver.spawn() sets the baremetal node status
and the nova instance status is also set properly. If an error occurs
during the deployment, an exception is raised within driver.spawn()
allowing nova to follow the normal cleanup and notify paths.

This also allows the baremetal PXE driver to delete cached image files
when a baremetal deployment fails.

Fixes bug 1088655.

Change-Id: I4feefd462fd956c9780995ec8b05b13e78278c8b
Better error handling in baremetal spawn & destroy

Improve the exception handling in baremetal driver spawn and destroy.

Adds unit tests for destroy to check that bm_node is disassociated
from instance_uuid only when destroy succeeds, and is set to ERROR state
(but still associated) if any exception is not handled.

Change-Id: I75853210d4883b352da9876a682dbb7288689541
Improve performance of baremetal list_instances.

This patch adds two new methods to baremetal/db/api:
 - bm_node_get_associated
 - bm_node_get_unassociated
which return a list of nodes either associated or not associated to a
Nova instance, as determined by the instance_uuid field, respectively.

Now, list_instances and get_available_nodes methods only fetch assocated
and unassociated nodes from the baremetal database, instead of fetching
all nodes.

Change-Id: Ib875d26d7b85a9dbf08c925a89802e2cd614e363
Identify baremetal nodes by UUID.

- add a new 'uuid' column to 'bm_nodes' table
- provide a new db/api method for accessing nodes by their uuid
- return this to Nova as the nodename / hypervisor_hostname

In this way, a baremetal node is uniquely identified to the user,
whether they use a per-compute baremetal db or global baremetal db.

It will also allow for an instance to be booted on a specific baremetal
node using a means akin to the current force-hosts (TBD in later patch).

Also, creates two new Exceptions to disambiguate between
InstanceNotFound and NodeNotFound / NodeNotFoundByUUID.

Change-Id: I81105a201588fdef31cffabdae260bb43017bcd1
Baremetal driver returns accurate list of instance

Add 'instance_name' to bm_nodes table so that baremetal driver is able
to return the names of all instances it believes are still running.

Previously, baremetal.driver.list_instances was fetching all allocated
instances from baremetal database, then calling VirtAPI to get the
instance name. This would raise an InstanceNotFound exception for
deleted instances. This prevented ComputeManager from ever detecting
a running-but-deleted baremetal instance, and could leave baremetal
instances in an undeletable state.

Fixes bug 1096723.

Change-Id: Ifae532e8e70e97e48c589608cb3c7000bb6a7609