Fixes Hyper-V volume discovery exception message

On some editions of Windows / Hyper-V server the SAN policy is set
by default to "Online All", bringing online any disk, local or shared,
attached to the host.

Since only offline disks can be attached as passthrough disks to a Hyper-V
VM, this prevents Cinder volumes from being attached to instances,
resulting in an exception that can be particularly hard to troubleshoot
without a clear knowledge of the context.

This patch includes some details in the exception message for
troubleshooting purposes.

Change-Id: I6d0b49812f8ca965ca53be34ef009997c8e79728
Closes-Bug: #1361419
Remove kombu as a dependency for Nova

It was required before oslo.messaging.

Change-Id: I3159b8509ac5b4020cd584c894eb131bb16df97c
Faster get_attrname in nova/objects/base.py

get_attrname function is called very very often,
and it's string concatenation method is not fast.

* Changing the string concatenation to a 2 times
  faster + operator instead of the % operator.

Change-Id: I326fd6dda8dd41a90757c69867afd62063231362
Refactor Ironic driver tests as per review comment

During the Ironic driver merge into Nova, there were some
suggestions on refactoring the tests as per this review
(https://review.openstack.org/#/c/111425/19/nova/...
...tests/virt/ironic/test_driver.py)

This patch implements these suggestions.

Change-Id: I05e8b3d01bffebe2f68bbd50fc775e1ef48616cc
Only load necessary instance info for use in sync power state

During sync power state, instance is refreshed from the db. If the
instance gets deleted, this can result in InstanceInfoCacheNotFound
being thrown due to info_cache being loaded and therefore refreshed.

Sync power state does not make use of any extra instance attrs, so
this problem can be avoided by not loading any extra attrs in the
first place.

Two callers of _sync_instance_power_state() are updated to not load
extra attrs. 1) The periodic sync and 2) the lifecycle event handler.

Change-Id: I150bd95da65263981608dd9a033232bd142e737f
Closes-Bug: #1367845
Don't modify columns_to_join formal parameter in _manual_join_columns

The _manual_join_columns method was modifying the columns_to_join list
passed to it which, if not copied by the original caller, can introduce
bugs in the calling code if the original columns_to_join list is re-used
later, e.g. making nova objects with an expected_attrs list that has
been mutated by the DB API call.

This change makes a copy of the original list and passes back the new
list for the DB API methods to use, and adds some documentation to
hopefully clarify intent and usage.

A test is added to demonstrate the expected behavior of the utility
method.

Closes-Bug: #1393491

Change-Id: I49511f608291b66642ced00a3c13833e18fab3d7
Remove unused network_api.get_instance_uuids_by_ip_filter()

Last usage was removed in ae781ee97947c33d6d43e4c21df4f338c875bf1c

Change-Id: I33dc14085ca0c887778c7379f0498714e8d30425
Improve error return code of floating_ips in v2/v2.1 api

1.change exception.InvalidID from 404 to 400 when passing a invalid
id.

2.handle exceptions when calling get_instance_id_by_floating_address()
  FloatingIpNotFoundForAddress -> HTTPNotFound
  FloatingIpMultipleFoundForAddress ->HTTPConflict

3.change HTTPUnprocessableEntity to HTTPConflict when remove a floating
ip which is not associated the instance

Change-Id: I0c9cd5b50a2d3cf024e5c4c3506ae2d0e6394c88
Add delete not found unit testcase for floating_ip api

This patch adds unit test case for the delete api of
floating_ip extension.
V2 and V21 have difference return code when delete
a invalid when given a invalid id of floating ip.

This patch addes both nova-network api and neutronv2 api
test cases.

Partially implements blueprint v2-on-v3-api
Change-Id: I40a5b133c98e01e8f5c613b765b8d821beec03a6
Set vm state error when raising unexpected exception in live migrate

The instance stuck at migrating task_state when unexpected exception.
This is confuse for user, this patch set the vm_state to error.

Change-Id: Ib1b97452bc5e777c66c4d368f71156dbe1e116b7
Partial-Bug: 1397153
Add shelve and unshelve info into devref doc

The diagram on http://docs.openstack.org/developer/nova/devref/vmstates.html
does not currently show the "SHELVED_OFFLOADED" state and the
associated "shelve", "shelveOffload" and "unshelve" state transitions.

Change-Id: I50eb07a84336160b8120ca294598576eeb982d28
Closes-Bug: #1394215
Error code for creating secgroup default rule

Trying to create secgroup group default rules
that already exists gives HttpBadRequest error,
but it should raise HttpConflict.

So updated the code to HttpConflict error.
Closes-Bug: #1374320

Change-Id: I93c6eb7fb96e7109404206367ed81495a4724296
VMware: allow selection of vSAN datastores

With support for handling of stream-optimized disk images added, the
vmware driver is now able to deploy to vSAN datastores, so add support
for the the 'vsan' datastore type, and update the datastore selection
logic to take this type into account when when spawning a
stream-optimized image.

Implements: blueprint vmware-vsan-support
Co-authored-by: Radoslav Gerganov <rgerganov@vmware.com>

Change-Id: Id43ad4adcad212a8e0d8ab3830af62d4afda1442
Libvirt: SMB volume driver

Currently, there are Libvirt volume drivers that support
network-attached file systems such as Gluster or NFS. This patch
adds a new volume driver in order to support attaching volumes
hosted on SMB shares.

Co-Authored-By: Gabriel Samfira <gsamfira@cloudbasesolutions.com>

DocImpact

Change-Id: I1db3d2a6d8ee94932348c63cc03698fdefff0b5c
Implements: blueprint libvirt-smbfs-volume-support
Compute: Catch binding failed exception while init host

While compute starts it will init all instances,
if an exception is raised from one instance
(e.g NovaException during plug_vifs), then the
compute process exits unexpectedly because of
this unhandled exception.
This commit changes the NovaException to more
appropriate VirtualInterfacePlugException and
catches it during init host, as well as the
instance is set to error state, with this change
the compute process can be started normally even
if this VirtualInterfacePlugException is raised.

Closes-bug: #1324041

Change-Id: Ia584dba66affb86787e3069df19bd17b89cb5c49
Clear libvirt test on LibvirtDriverTestCase

We already create a class variable for connection,
no need to create each one for every test cases.

Change-Id: I635f5993888b21dc9b497e588057af5f794f55b3
Compute Add build_instance hook in compute manager

The build_instance hook provides a hook close to the actual spawn
of an instance. It distinguishes itself from the create_instance
hook because, in the case of a successful build, the instance will
exist on the hypervisor when the build_instance post hook is
called, instead of simply just being scheduled as is the case with
the create_instance post hook. This is useful as some details may
not be available until after scheduling.

Co-Authored-By: Corey Wright <corey.wright@rackspace.com>
Change-Id: Ic12abf6315bbc6b010b8222ef296ec65cb4b4f95
Use a copy of NEW_NETWORK for test_networks

In test_networks, NEW_NETWORK is used as a request body between unit
tests. "Create a network" API changes a request body internally, so
each test should use a copy of NEW_NETWORK to avoid affecting to the
other unit tests. However one test uses NEW_NETWORK directly.
This patch makes each test use a copy of NEW_NETWORK for avoiding
similar problem in the future.

Change-Id: I9e12fe89956d18dde54bec83c15af258d66e0682
Add a validation format "cidr"

Nova accepts CIDR attributes on network REST APIs and checks its format
on each API. On v2.1 API, the attributes will be checked with JSONSchema.
In addition, Neutron contains its own "cidr" check for its REST APIs.
This patch add a validation format "cidr" by porting Neutron's check from
https://github.com/openstack/neutron/blob/master/neutron/api/v2/attributes.py#L293

Partially implements blueprint v2-on-v3-api

Change-Id: I03601f8fa34fe7617e4f963cdfd4b178eae36565
Share server access ips tests between V2 & V2.1

As server access ips should behave same in V2 and V2.1,
access ips tests can be shared among V2 & V2.1 to cleanup
duplicated test code.

This patch share those tests between V2 & V2.1

Partially implements blueprint v2-on-v3-api

Change-Id: Ife0b3f35ec6bd0d3c0052d2a8c0f314638a07ca3