Fixes Hyper-V volume discovery exception message

On some editions of Windows / Hyper-V server the SAN policy is set
by default to "Online All", bringing online any disk, local or shared,
attached to the host.

Since only offline disks can be attached as passthrough disks to a Hyper-V
VM, this prevents Cinder volumes from being attached to instances,
resulting in an exception that can be particularly hard to troubleshoot
without a clear knowledge of the context.

This patch includes some details in the exception message for
troubleshooting purposes.

Change-Id: I6d0b49812f8ca965ca53be34ef009997c8e79728
Closes-Bug: #1361419
Only load necessary instance info for use in sync power state

During sync power state, instance is refreshed from the db. If the
instance gets deleted, this can result in InstanceInfoCacheNotFound
being thrown due to info_cache being loaded and therefore refreshed.

Sync power state does not make use of any extra instance attrs, so
this problem can be avoided by not loading any extra attrs in the
first place.

Two callers of _sync_instance_power_state() are updated to not load
extra attrs. 1) The periodic sync and 2) the lifecycle event handler.

Change-Id: I150bd95da65263981608dd9a033232bd142e737f
Closes-Bug: #1367845
Rename private functions in db.sqla.api

instance_get_all_by_filters() called 3 helper functions: exact_filter,
regex_filter and tag_filter. None of these functions had any other
callers.

This change prefixes them with _ to identify them as private.
Additionally, it removes the model arguments to the functions as these
always relate to an Instance and/or its InstanceMetadata.
Consequently, the functions are renamed to identify them as specific
to an Instance query. exact_filter is moved from the top of the module
to be next to the other 2.

Change-Id: I671315122438022b13d6b315b577ef2a5cf42545
Don't modify columns_to_join formal parameter in _manual_join_columns

The _manual_join_columns method was modifying the columns_to_join list
passed to it which, if not copied by the original caller, can introduce
bugs in the calling code if the original columns_to_join list is re-used
later, e.g. making nova objects with an expected_attrs list that has
been mutated by the DB API call.

This change makes a copy of the original list and passes back the new
list for the DB API methods to use, and adds some documentation to
hopefully clarify intent and usage.

A test is added to demonstrate the expected behavior of the utility
method.

Closes-Bug: #1393491

Change-Id: I49511f608291b66642ced00a3c13833e18fab3d7
Remove old Baremetal Host Manager

Baremetal was deprecated in Juno and its driver was removed in
46ed619b9ab1d61582f36155eea0d4a88f31fd50

This cleans out the parts that's still left in the scheduler.

Change-Id: I86f583f7d0747a164dae4530e25f8b01c2144673
Set vm state error when raising unexpected exception in live migrate

The instance stuck at migrating task_state when unexpected exception.
This is confuse for user, this patch set the vm_state to error.

Change-Id: Ib1b97452bc5e777c66c4d368f71156dbe1e116b7
Partial-Bug: 1397153
Add shelve and unshelve info into devref doc

The diagram on http://docs.openstack.org/developer/nova/devref/vmstates.html
does not currently show the "SHELVED_OFFLOADED" state and the
associated "shelve", "shelveOffload" and "unshelve" state transitions.

Change-Id: I50eb07a84336160b8120ca294598576eeb982d28
Closes-Bug: #1394215
Remove the volume api related useless policy rules

There are some old policy rules used by old nova volume, those are
useless anymore, this patch clean them up.

Change-Id: Ibcdba00652808158019600f8ad75e80c85f9423e
Closes-Bug: #1398686
VMware: allow selection of vSAN datastores

With support for handling of stream-optimized disk images added, the
vmware driver is now able to deploy to vSAN datastores, so add support
for the the 'vsan' datastore type, and update the datastore selection
logic to take this type into account when when spawning a
stream-optimized image.

Implements: blueprint vmware-vsan-support
Co-authored-by: Radoslav Gerganov <rgerganov@vmware.com>

Change-Id: Id43ad4adcad212a8e0d8ab3830af62d4afda1442
Libvirt: SMB volume driver

Currently, there are Libvirt volume drivers that support
network-attached file systems such as Gluster or NFS. This patch
adds a new volume driver in order to support attaching volumes
hosted on SMB shares.

Co-Authored-By: Gabriel Samfira <gsamfira@cloudbasesolutions.com>

DocImpact

Change-Id: I1db3d2a6d8ee94932348c63cc03698fdefff0b5c
Implements: blueprint libvirt-smbfs-volume-support
Compute: Catch binding failed exception while init host

While compute starts it will init all instances,
if an exception is raised from one instance
(e.g NovaException during plug_vifs), then the
compute process exits unexpectedly because of
this unhandled exception.
This commit changes the NovaException to more
appropriate VirtualInterfacePlugException and
catches it during init host, as well as the
instance is set to error state, with this change
the compute process can be started normally even
if this VirtualInterfacePlugException is raised.

Closes-bug: #1324041

Change-Id: Ia584dba66affb86787e3069df19bd17b89cb5c49
Compute Add build_instance hook in compute manager

The build_instance hook provides a hook close to the actual spawn
of an instance. It distinguishes itself from the create_instance
hook because, in the case of a successful build, the instance will
exist on the hypervisor when the build_instance post hook is
called, instead of simply just being scheduled as is the case with
the create_instance post hook. This is useful as some details may
not be available until after scheduling.

Co-Authored-By: Corey Wright <corey.wright@rackspace.com>
Change-Id: Ic12abf6315bbc6b010b8222ef296ec65cb4b4f95
Assert order of DB index members

Order of columns can matter. Most SQL databases can use the leading
columns for optimizing queries that don't include all of the columns.
Ignoring the order can result in an index being added that may not
be able to be used to optimize the same queries.

Change-Id: Ie827aaacbcfbc01e32591d29aab79ef75aa0c5d9
Adds global API version check for microversions

Adds a check for a request that the version requested is within the
global API version range supported by the REST API. Both the minimum
and maximum are currently set to "2.1". The maximum will be increased
everytime an API change is made.

Also sets up some template/doc files for clearly documenting
the REST API changes with each microversion increment.

Partially implements blueprint api-microversions

Change-Id: Ie7fdb2928d957c03ed788c2ddd29fe798c645fce
Use a copy of NEW_NETWORK for test_networks

In test_networks, NEW_NETWORK is used as a request body between unit
tests. "Create a network" API changes a request body internally, so
each test should use a copy of NEW_NETWORK to avoid affecting to the
other unit tests. However one test uses NEW_NETWORK directly.
This patch makes each test use a copy of NEW_NETWORK for avoiding
similar problem in the future.

Change-Id: I9e12fe89956d18dde54bec83c15af258d66e0682
Add a validation format "cidr"

Nova accepts CIDR attributes on network REST APIs and checks its format
on each API. On v2.1 API, the attributes will be checked with JSONSchema.
In addition, Neutron contains its own "cidr" check for its REST APIs.
This patch add a validation format "cidr" by porting Neutron's check from
https://github.com/openstack/neutron/blob/master/neutron/api/v2/attributes.py#L293

Partially implements blueprint v2-on-v3-api

Change-Id: I03601f8fa34fe7617e4f963cdfd4b178eae36565
Workflow documentation is now in infra-manual

Replace URLs for workflow documentation to appropriate parts of the
OpenStack Project Infrastructure Manual.

Change-Id: Ice00abcbb431b29e28beb0cd8810b8653ef66139
Share server access ips tests between V2 & V2.1

As server access ips should behave same in V2 and V2.1,
access ips tests can be shared among V2 & V2.1 to cleanup
duplicated test code.

This patch share those tests between V2 & V2.1

Partially implements blueprint v2-on-v3-api

Change-Id: Ife0b3f35ec6bd0d3c0052d2a8c0f314638a07ca3
Remove except Exception cases

Replaced 'except Exception' cases with the specific exceptions
that the given code can throw in the util module. In some cases
we added extra unit test to cover the specific exception.

Change-Id: I980237ed9d27f122b4725b8c05015a81764dd6b8
Partial-Bug: #1223605
Check for floating IP pool in nova-network

There are no checks in nova-network to see if a specified
floating IP pool exists or not before attempting to allocate
an IP address. If a pool does not exist, nova throws an
unrelated error - 'No more floating ips available'. The
following patch adds a check for the existence of the IP pool
and throws a FloatingIpPoolNotFound if it is not found.

Change-Id: I72cb8dd044f7ac094287bad7df9167b8e77a4b5b
Related-Bug: #1355623