libvirt: Avoid Glance.show on hard_reboot

When a Libvirt host is rebooted, we can optionally choose to automatically
spin back up the instances using the `resume_state_on_host_boot` call. This,
in turn, uses `_hard_reboot` to bring back the instances.

The problem is that `_get_guest_xml` which `_hard_reboot` is using is always
making a call to Glance.show since `image_meta` isn't being passed into it. In
addition to this being an extra latency-heavy call, this is big problem
because the request is *server-generated* not *user-generated* so we won't
have the necessary user-request context to make the Glance call.

In the absense of a general user-impersonation mechanism, the current
workaround is to use cached image-metadata, which works in this case as well.

So the fix is to pass that `image_meta` that we already pull from instance
metadata in `_hard_reboot` and pass it into `_get_guest_xml` and thus avoid
the extra request to Glance.

Change-Id: I2203ac709405ee784ee5ec017aa475575a46a0df
Closes-Bug: 1339386
Adds more policy control to cells ext

Allow create,delete,update and sync_instances to
have fine grained policy control.

UpgradeImpact
DocImpact

Change-Id: I2e32ae2f37d3b599585d25535c79eecf6485b462
Closes-Bug: 1335901
VMware: power_off_instance support

Commit 1e1915aaaca38b5691794e0e052a42b9d95dd3c2 introduced a new
utility method to shutdown a VM. This will now be moved to vm_util
similar to refactoring in blueprint vmware-spawn-refactor.

This patch does the following:
1. Creates a vm_util method - power_off_instance.
2. Corrects the unit test for the unrescue. That did not correctly validate
   the correct vm ref for the power off
3. Added specific tests for the new method that has been added
4. Drops the test test_power_off_suspended as this is not a valid test
   case.

Change-Id: I6b7c260488bbd0278f6e7ce98be77e958f1576ab
Fix nova cells exiting on db failure at launch

We have seen cases where db errors at launch can cause cell
services to exit without retrying.  The service shouldn't
exit __init__ much like it doesn't later by handling
this type of exceptions.  We'll wait an hour and
then give up.

Change-Id: I24c9eb811d50d1fa6a5e4a5f595ebf68ded3b7b5
Closes-Bug: 1342257
Check for resize path on libvirt instance delete

If an instance is deleted after the instance's disk image path has
been renamed by adding the "_resize" suffix to it but before the
resize operation completes, the libvirt driver will not delete the
orphaned files and manual intervention is needed to get them deleted.

This fix addresses the issue by attempting to rename the instance path
by adding a "_del" suffix and if that fails, renaming the instance path
with the "_resize" suffix by replacing the "_resize" suffix with the
"_del" suffix.  If both renaming operations fail, the sequence is
repeated, in case the the disk image path initially had the "_resize"
suffix and another thread removed it before the second rename operation
was attempted. These rename operations are used in favor of checking
for the existence of paths and deleting if found, because rename
operations are atomic whereas another thread could rename the path
between the exist check and the deleting.

Regardless of the outcome of the renaming operations, the existence of
the instance path with the "_del" suffix is verified and if it exists,
it is deleted. This is done in case a prior delete operation that
managed to create the "_del" path was subsequently interrupted before
all instance files could be deleted.

Note that the LibvirtConnTestCase.test_delete_instance_files test case
was removed in order to eliminate redundancy.

Closes-Bug: #1308565

Change-Id: Ifcb2e18211347ccf3e5472779c5917a729a6eced
Turn periodic tasks off in all unit tests

We shouldn't rely on periodic tasks to randomly run in unit tests,
instead if we need them to run we should explicitly call them out.

Change-Id: I35850c47288e50d3146771dc7a9a54d62e36c3cf
Related-Bug: 1311778
VMware: do not cache image when root_gb is 0

Save space on datastore when the instance size has root_gb of
0. There is no need to cache the disk as it already exists on
the datastore.

TrivialFix

Change-Id: Iac75d97f40a1a91b8fa2df7fb4318fc2965fd93a
Closes-bug: #1304593
Convert to importutils

We had some remaining users of __import__ that really deserved to
be converted to oslo's importutils.

Change-Id: I8c0c3e885151018b974025087a4abc96dd6ae77c
Add quota limit create/update methods to Quotas object

This adds two methods to the Quotas object that are effectively
passthroughs to the DB api calls. Since the Quotas object isn't
actually a representation of what is in the database, and because
we've (incorrectly) got some logic in the DB API around how the
quotas are applied, this is just a temporary step. Ideally, we
would (will) refactor Quotas in several ways in the near future,
at which time the way all of this works can be revisited and
cleaned up.

Related to blueprint compute-manager-objects-juno

Change-Id: If93614cd27e56667e0f106e8045dd1adc633ff1b
Revert "Add extensible resources to resource tracker"

This bug added a regression to both nova-bm and ironic,
neither can deploy instances.
Fixes-bug: #1347795

This reverts commit 50b4ba4ee583d25eef10a6608172c002f9bec6f2.

Change-Id: Icc8d629467911972480b633c7808a0964c9f1c7d
Raise NotImplemented default-security-group-rule api with neutron

Neutron currently does not implement the nova extension
default-security-group-rule which allows an admin to set which rules are in
the default security group when a tenant is onboarded. Implementing this
type of functionality should be done in neutron directly as proxying the
api calls to do this from nova is not possible as neutron creates the
default security group itself. Previously, if one tried to use this api
with neutron a 500 error was raised, now a 501 error is raised.

Change-Id: I6d1bb1c9c2d79278a85b96aea800b4b1f3489225
Closes-bug: 1326958
Augment oslo's default log levels with nova specific ones.

Thanks to I49a82c5b3446784d254ca558f51b6c3e5c2028eb, we can augment
oslo's set of default log levels with project specific ones.

Add keystonemiddleware as warn level since its verbose and nova isn't
trying to debug keystonemiddleware.

Add routes.middleware since this generates 3 log entries per API request
and doesn't help make debugging easier.

Change-Id: Ib5164a22c5cbfa4f9d881b97bb1f623cd4d2f3f3
Merge "Raise NotImplemented default-security-group-rule api with neutron"
Merge "Check for resize path on libvirt instance delete"
reduce network down time during live-migration

Called unplug_vifs() method before post_live_migration_to_destination call
because floating ip address will not work until the vifs are unplugged
from the source compute node.

Added new method post_live_migration_at_source in virt driver and
implemented it in libvirt driver to unplug the vifs at source. Other
drivers will raise NotImplementedError.
Added new keyword argument destroy_vifs to cleanup() method to verify
if unplug_vifs() is already called then do not call it again.

Closes-Bug: #1284996
Change-Id: Ida3ed3bef77239e5a0fdbf8866e8a4421d4f8222
Add Hyper-V driver in the "compute_driver" option description

The description of the option "compute_driver" should include
hyperv.HyperVDriver along with the other supported driver.

Change-Id: Ic7e16fd5154609987b2ef0b6f8ee52619ccf489c
Closes-bug: #1347777
Merge "Convert to importutils"
Merge "Adds more policy control to cells ext"
Merge "VMware: do not cache image when root_gb is 0"
Fix and Gate on E265

pep8 E265 makes sure block comment start with '# '. Fix and gate on this
new rule as it helps improve comment readability.

In the few cases where the comment was just commented out code, remove
the comment.

Change-Id: Iea1c445df8ddc2b6c17a4ab697ad756eef2f91fa