libvirt: Avoid Glance.show on hard_reboot

When a Libvirt host is rebooted, we can optionally choose to automatically
spin back up the instances using the `resume_state_on_host_boot` call. This,
in turn, uses `_hard_reboot` to bring back the instances.

The problem is that `_get_guest_xml` which `_hard_reboot` is using is always
making a call to Glance.show since `image_meta` isn't being passed into it. In
addition to this being an extra latency-heavy call, this is big problem
because the request is *server-generated* not *user-generated* so we won't
have the necessary user-request context to make the Glance call.

In the absense of a general user-impersonation mechanism, the current
workaround is to use cached image-metadata, which works in this case as well.

So the fix is to pass that `image_meta` that we already pull from instance
metadata in `_hard_reboot` and pass it into `_get_guest_xml` and thus avoid
the extra request to Glance.

Change-Id: I2203ac709405ee784ee5ec017aa475575a46a0df
Closes-Bug: 1339386
object-ify availability_zones

This change converts availability_zones to use the Aggregate and
Service objects instead of direct database access.

Related to blueprint compute-manager-objects-juno

Change-Id: I2ec7373f063ed728dde2af3b7c8259f4391885bc
Restore instance flavor info when driver finish_migration fails

when instance resize, it will call finish_migration at last to create
new instance and destroy old instance. If driver layer has problem in
create new instance, the instance will be set to 'ERROR' state.

we are able to use reset-state --active to reset the instance and use it
but the instance information is set to new flavor
and not reverted to old one

Change-Id: I5961260f50e7893c7bf03e329e06edfd4f295640
Closes-Bug: #1296519
Store volume backed snapshot in current tenant.

Fix owner of a creating volume backed snapshot.

Snapshot of an instance booted on a volume based on another tenant's
public image is created in the wrong tenant when invoked by admin.

Snapshot metadata (including owner) is based on image metadata. But
when the snapshot is being created by admin, Glance doesn't change
it's owner if it's set. So we forcibly remove owner (tenant)
attribute from image metadata.

Change-Id: I662dfa4f81e24cb2553ffa2578f4c8530eee9fd3
Closes-Bug: #1322195
Adds more policy control to cells ext

Allow create,delete,update and sync_instances to
have fine grained policy control.

UpgradeImpact
DocImpact

Change-Id: I2e32ae2f37d3b599585d25535c79eecf6485b462
Closes-Bug: 1335901
Partial oslo-incubator sync

Generated with:
python update.py --base nova --dest-dir ~/Develop/tmp/nova/ --modules
timeutils,sslutils,importutils,memorycache,strutils,context,fileutils,versionutils,systemd

context:
 9b73877 Add a RequestContext.from_dict method
 85d1ce6 Python 3: enable tests/unit/middleware/test_request_id.py
fileutils:
  2b966f9 Fix deletion of cached file for policy enforcer
gettextutils:
 3d90045 Backport code for i18n to check lazy at runtime
jsonutils:
 ef37e03 Added missing jsonutils.dump() function
log:
 5cac11d Merge "Add default log level for websocket"
 433fa0b Make logging_context_format_string optional in log.set_defaults
 ac92c06 Add default log level for websocket
 0aa2bd4 Merge "Ability to customize default_log_levels for each project"
 5fd77eb Ability to customize default_log_levels for each project
 i4d9328c Python 3: enable tests/unit/test_log.py
 722f418 Merge "update new requests logger to default WARN"
 cb5a804 Move `mask_password` to strutils
memorycache:
 90ae24b Remove redundant default=None for config options
 297d772 Raise exception when importing memcache error
strutils:
 cb5a804 Move `mask_password` to strutils
systemd:
 17c4e21 Fix docstring indentation in systemd
 667d1ba Fixed spelling error - occured to occurred
versionutils:
 a2ad3a2 Allow deprecated decorator to specify no plan for removal
 05ae498 Add JUNO as a target to versionutils module

Change-Id: I9e8e8e9b15075d99cf394170c6f5300e7bd0d4cc
Correctly reject request to add lists of hosts to an aggregate

Attempting to add or remove a list of hosts to an aggregate
results in an internal server error. This patch fixes the input
validation so such a request is correctly rejected and a 400
error returned.

Change-Id: I760db36d685cff20b01a72160b8b3fbb8a70b412
Closes-Bug: #1321653
Remove warn log for over quota

Over quota exception is already raised at API layer
and no need to print a warning log.

Change-Id: If44d5f2dd0e11d15d57cf6ab247f5e9e7fe75a9c
Fix error status code for agents

When passing bad body in a request, most APIs return BadRequest response.
However, agents API doesn't do it.
This patch fixes the error status code and adds a unit test related to
this change.

This patch's idea most came from
Ken'ichi Ohmichi's patch https://review.openstack.org/#/c/107266/

Change-Id: I9602183ade4d3c0105e2498aebb7e33eb72f86f0
Add differencing vhdx resize support in Hyper-V Driver

Differencing VHDX images can be resized, unlike differencing VHD images.
Even so, the Nova Hyper-V driver currently does not support this.

This feature is required for resizing existing instances which use CoW
VHDX images and also in order to resize the root disk image when
spawning a new instance.

The methods which get the internal maximum size of a vhd/vhdx
image have been updated to lookup the parent disk and get the according
size instead of raising an exception in case of differencing images.

Change-Id: I867a9b35e5273b96afc443e5c436a15d0c45161b
Implements: blueprint add-differencing-vhdx-resize-support
Add extensible resources to resource tracker

A resource plugin extension point is added to the resource
tracker to allow the types of resources allocated at the compute
node to be extensible. Information maintained by these plug-ins
is written to the compute_nodes table in the database. The
scheduler uses the information in the compute_nodes table to
determine scheduling decisions.

A plugin that implements vcpu resource tracking is included
and all other code for tracking vcpu has been removed. This
example ensures the plugins are tested in gate jobs.

Co-Authored-By: Andrea Rosa  <andrea.rosa@hp.com>
Co-Authored-By: Paul Murray  <pmurray@hp.com>
This is part of: blueprint extensible-resource-tracking

Change-Id: I64108338e3c958ba1276aaf113a68861cbe286f5
VMware: power_off_instance support

Commit 1e1915aaaca38b5691794e0e052a42b9d95dd3c2 introduced a new
utility method to shutdown a VM. This will now be moved to vm_util
similar to refactoring in blueprint vmware-spawn-refactor.

This patch does the following:
1. Creates a vm_util method - power_off_instance.
2. Corrects the unit test for the unrescue. That did not correctly validate
   the correct vm ref for the power off
3. Added specific tests for the new method that has been added
4. Drops the test test_power_off_suspended as this is not a valid test
   case.

Change-Id: I6b7c260488bbd0278f6e7ce98be77e958f1576ab
Fix nova cells exiting on db failure at launch

We have seen cases where db errors at launch can cause cell
services to exit without retrying.  The service shouldn't
exit __init__ much like it doesn't later by handling
this type of exceptions.  We'll wait an hour and
then give up.

Change-Id: I24c9eb811d50d1fa6a5e4a5f595ebf68ded3b7b5
Closes-Bug: 1342257
Check for resize path on libvirt instance delete

If an instance is deleted after the instance's disk image path has
been renamed by adding the "_resize" suffix to it but before the
resize operation completes, the libvirt driver will not delete the
orphaned files and manual intervention is needed to get them deleted.

This fix addresses the issue by attempting to rename the instance path
by adding a "_del" suffix and if that fails, renaming the instance path
with the "_resize" suffix by replacing the "_resize" suffix with the
"_del" suffix.  If both renaming operations fail, the sequence is
repeated, in case the the disk image path initially had the "_resize"
suffix and another thread removed it before the second rename operation
was attempted. These rename operations are used in favor of checking
for the existence of paths and deleting if found, because rename
operations are atomic whereas another thread could rename the path
between the exist check and the deleting.

Regardless of the outcome of the renaming operations, the existence of
the instance path with the "_del" suffix is verified and if it exists,
it is deleted. This is done in case a prior delete operation that
managed to create the "_del" path was subsequently interrupted before
all instance files could be deleted.

Note that the LibvirtConnTestCase.test_delete_instance_files test case
was removed in order to eliminate redundancy.

Closes-Bug: #1308565

Change-Id: Ifcb2e18211347ccf3e5472779c5917a729a6eced
Rename virtutils to the more common libvirt_utils

`libvirt_utils` is the more prevalent alias in the code for the module, so
rename `virtutils` to match.

Change-Id: Ib8d4291866bc53fa1980b70dd0f12e6b1fe10436
Turn periodic tasks off in all unit tests

We shouldn't rely on periodic tasks to randomly run in unit tests,
instead if we need them to run we should explicitly call them out.

Change-Id: I35850c47288e50d3146771dc7a9a54d62e36c3cf
Related-Bug: 1311778
VMware: do not cache image when root_gb is 0

Save space on datastore when the instance size has root_gb of
0. There is no need to cache the disk as it already exists on
the datastore.

TrivialFix

Change-Id: Iac75d97f40a1a91b8fa2df7fb4318fc2965fd93a
Closes-bug: #1304593
Convert to importutils

We had some remaining users of __import__ that really deserved to
be converted to oslo's importutils.

Change-Id: I8c0c3e885151018b974025087a4abc96dd6ae77c
Raise specific error of network IP allocation

Currently Nova raise "Neutron error: quota exceeded" when
neutron client state code is 409 in IP allocation without
caring about if the real error is over quota.

This fixes raises PortLimitExceeded if error from neutron
client is OverQuota and raises NoMoreFixedIps if error from
neutron client is IpAddressGenerationFailure. If the error
is neither of these two errors above, it reraises the original
exception from neutron client.

Change-Id: I0d1942587a359acccfca5298c2dcb4b21d971a42
Closes-Bug: 1331353
libvirt: add parsing of NUMA topology in capabilities XML

Introduce classes to nova.virt.libvirt.config for parsing
the NUMA topology information in the capabilities XML
schema.

Blueprint: virt-driver-numa-placement
Change-Id: I4948514cf38268663cdcf4bb31f335a19a7ec9d1