Implement methods to modify volume metadata.

There are four methods in nova/volume/cinder.py which are
NotImplemented. They are as follows:
  1. get_volume_metadata
  2. delete_volume_metadata
  3. update_volume_metadata
  4. get_volume_metadata_value
These methods are required in cases where nova needs to modify
a cinder volume's metadata, e.g. attach and detach time. It also
completes the cinder.API class.

Change-Id: I360076ae28db43e661466f556425947813c9040e
Closes-Bug: #1296164
Keep Migration status in automatic confirm-resize

function in compute/manager.py _poll_unconfirmed_resizes
will translate the migrate status from finished to error
whenever finds a problem ,Consider the following case

1) _poll_unconfirmed_resizes runs and founds several migrations to be confirmed
2) user deletes the instance, so its task state will be changed to DELETING
3) _poll_unconfirmed_resizes will find the task state is not None,
it will make the migration error status
4) following code in _delete
if instance.vm_state == vm_states.RESIZED:
self._confirm_resize_on_deleting(context, instance)
will fail because migration status already updated

User is able to do the migration operations when they found error
so we don't need to handle this kind of error for them

Change-Id: Ia79dd56c9c1fa8add7dc138d838cbfbac4523ac3
Closes-Bug: #1301696
Issue multiple SQL statements in separate engine.execute() calls

Some sqlalchemy drivers (eg: mysqlconnector) don't support
engine.execute("sql stmt 1; sql stmt 2;") and require a separate
execute() call for each SQL statement.  After discussions with
sqlalchemy author, he confirmed it would be better to fix callers rather
than attempt to patch in support for multiple statements.

With this patch, nova-manage db sync succeeds using mysqlconnector.

Change-Id: I57e6ecdafe90110eaffe757f0ef0bf7b41b0f3e3
Add graphviz to list of distro packages to install

Commit a507d42cf5d9912c2b3622e84afb8b7d3278595b makes the
doc builds fail on warnings, so people are probably going
to be running 'tox -e docs' more often. To run that you
need the graphviz package from the distro, so add it to
the list of packages needed when setting up a development
environment.

Change-Id: I47ffe63fddfbbd73c3fc698490ad07235caaf459
Closes-Bug: #1356983
VMware: Create VMwareImage object for image metadata

This change pulls the nested function _get_image_properties() out of
spawn() and makes it a factory method for the new VMwareImage object.

Linked clone logic is moved from a separate function in vmops.py to
the new get_image_properties() function.

vmware_images.get_vmdk_size_and_properties() is no longer used, and is
removed.

Tests have been updated and/or moved as appropriate. No tests have
been removed.

This change has been split out of
https://review.openstack.org/#/c/87002/, which was written by Shawn
Hartsock.

partial blueprint vmware-spawn-refactor

Co-authored-by: Shawn Hartsock <hartsocks@vmware.com>
Change-Id: I580f173da798318d2675c7c70bbdd19b266259f4
VMware: remove Host class

The class was only used by the ESX driver which was remove in the
commit 1deb31f85a8f5d1e261b2cf1eddc537a5da7f60b.

Change-Id: I026e4c0f463e78e33592c6408d2e688c74a0714b
VMware: remove specific VC support from class VMwareVolumeOps

This support is no longer needed as we only support the VC driver
since commit 1deb31f85a8f5d1e261b2cf1eddc537a5da7f60b.

Change-Id: Iad71fceb580620b1843ce3cf5d926343d8113945
Treat instance like an object in _default_block_device_names

The _default_block_device_names method already gets an instance object
but was treating it like a primitive dict so this change enforces the
object usage with dot notation when accessing fields.

Also updates the existing unit tests to use an instance object.

Part of blueprint compute-manager-objects-juno

Change-Id: I2706fa081ba427956dc4f643d1e5664eaa2573ed
Make usage_from_instances consider current usage

VirtNUMAHostTopology.usage_from_instances will now consider the current
usage of the passed host, and also accepts a 'free' keyword argument,
which if set to True, will cause the instance usage to be taken out of
the considered host. Also adds better handling of  None and emtpy
params.

Blueprint: virt-driver-numa-placement
Change-Id: I62c59b5e1dcbd4db247282f170c6f2f501bafd3d
Fixes Hyper-V resize down exception

The Hyper-V driver does not support resize down and is currently
raising an exception if the user attempts to do that, causing the
instance to go in ERROR state.

The driver should use the recently introduced instance faults
"exception.InstanceFaultRollback" instead, which will leave the
instance in ACTIVE state as expected.

Closes-Bug: #1354448

Change-Id: Ibaf8482562094cd2b3165dc62a907fa9e0e56e19
objects: Make use of utils.convert_version_to_tuple()

Make use of the convert_version_to_tuple() utility in objects that had
inlined an implementation of the same thing.

Change-Id: If48df452baa79eeb52218fcbad0e388edf8fd416
Power off commands should give guests a chance to shutdown

Currently in libvirt operations which power off an instance such as stop,
shelve, rescue, and resize simply destroy the underlying VM. Some
GuestOS's do not react well to this type of power failure, and so it would
be better if these operations followed the same approach as soft_reboot
and give the guest as chance to shutdown gracefully.

The shutdown behavior is defined by two values:

- shutdown_timeout defines the overall period a Guest is allowed to
complete it's shutdown.  The default valus is set via nova.conf and can be
overridden on a per image basis by image metadata allowing different types
of guest OS to specify how long they need to shutdown cleanly.

- shutdown_retry_interval defines how frequently within that period
the Guest will be signaled to shutdown.  This is a protection against
guests that may not be ready to process the shutdown signal when it
is first issued. (e.g. still booting).  This is defined as a constant.

This is one of a set of changes that will eventually expose the choice
of whether to give the GuestOS a chance to shutdown via the API.

This change implements the libvirt changes to power_off() and adds
a clean shutdown to compute.manager.stop().

Subsequent patches will:
- Add clean shutdown to Shelve
- Add clean shutdown to Rescue
- Convert soft_reboot to use the same approach
- Expose clean shutdown via rpcapi
- Expose clean shutdown via API

Partially-Implements: blueprint user-defined-shutdown
Closes-Bug: #1196924
DocImpact

Change-Id: I432b0b0c09db82797f28deb5617f02ee45a4278c
Resize should give guests a chance to shutdown

Currently in libvirt operations which power off an instance such as stop,
shelve, rescue, and resize simply destroy the underlying VM. Some
GuestOS's do not react well to this type of power failure, and so it would
be better if these operations followed the same approach as soft_reboot
and give the GuestOS as chance to shutdown gracefully.

This is one of a set of changes that will eventually expose the choice
of whether to give the GuestOS a chance to shutdown via the API.

This change implements the compute manager changes to resize for libvirt.

Partially-Implements: blueprint user-defined-shutdown
DocImpact

Change-Id: I03a52cfcc77d89e0982fcdf88be0651b3f12cbdd
Rescue should give guests a chance to shutdown

Currently in libvirt operations which power off an instance such as stop,
shelve, rescue, and resize simply destroy the underlying VM. Some
GuestOS's do not react well to this type of power failure, and so it would
be better if these operations followed the same approach as soft_reboot
and give the GuestOS as chance to shutdown gracefully.

This is one of a set of changes that will eventually expose the choice
of whether to give the GuestOS a chance to shutdown via the API.

This change implements the compute manager changes to rescue.

Partially-Implements: blueprint user-defined-shutdown
DocImpact

Change-Id: I296397f4fe3308be3727ec127929cea60d46b882
Shelve should give guests a chance to shutdown

Currently in libvirt operations which power off an instance such as stop,
shelve, rescue, and resize simply destroy the underlying VM. Some
GuestOS's do not react well to this type of power failure, and so it would
be better if these operations followed the same approach as soft_reboot
and give the GuestOS as chance to shutdown gracefully.

This is one of a set of changes that will eventually expose the choice
of whether to give the GuestOS a chance to shutdown via the API.

This change implements the compute manager changes to shelve.

Partially-Implements: blueprint user-defined-shutdown
DocImpact

Change-Id: Iec3dfd17725440958aac395ebc471e51afd6522e
Fixes Hyper-V unit test path separator issue

Fixes an issue in a unit tests were path separators were not
handled in a platform independent way causing the test to fail
on Windows.

Change-Id: Ifca77a561e5a95af8655f2373af665292b2ca3cb
Co-Authored-By: Zsolt Dudas <zdudas@cloudbasesolutions.com>
Closes-Bug: #1358702
Use rfc3986 library to validate URL paths and URIs

More work needs to be done in rfc3986 to give the user more control over what
they consider to be a valid URI in the context of RFC 3986. For example, a
previous incarnation of these tests checked that "1" and "abc" were invalid
when according to the RFC they are.

Update the API samples and tests to use valid URIs

DocImpact

Change-Id: I288fbaead64990db1053b7a11e82904611b8498f
Merge "Issue multiple SQL statements in separate engine.execute() calls"
Merge "Make usage_from_instances consider current usage"
Extend the docstring for obj_make_compatible() with examples

This adds to the docstring for obj_make_compatible() to help outline
the rules and scenarios to help developers understand what they need
to do in that method, and when.

Change-Id: Ifa8cb9a4f9a1a4d5afd0da4629a7d476a52c62d6