Fixes Hyper-V volume discovery exception message

On some editions of Windows / Hyper-V server the SAN policy is set
by default to "Online All", bringing online any disk, local or shared,
attached to the host.

Since only offline disks can be attached as passthrough disks to a Hyper-V
VM, this prevents Cinder volumes from being attached to instances,
resulting in an exception that can be particularly hard to troubleshoot
without a clear knowledge of the context.

This patch includes some details in the exception message for
troubleshooting purposes.

Change-Id: I6d0b49812f8ca965ca53be34ef009997c8e79728
Closes-Bug: #1361419
Only load necessary instance info for use in sync power state

During sync power state, instance is refreshed from the db. If the
instance gets deleted, this can result in InstanceInfoCacheNotFound
being thrown due to info_cache being loaded and therefore refreshed.

Sync power state does not make use of any extra instance attrs, so
this problem can be avoided by not loading any extra attrs in the
first place.

Two callers of _sync_instance_power_state() are updated to not load
extra attrs. 1) The periodic sync and 2) the lifecycle event handler.

Change-Id: I150bd95da65263981608dd9a033232bd142e737f
Closes-Bug: #1367845
Don't modify columns_to_join formal parameter in _manual_join_columns

The _manual_join_columns method was modifying the columns_to_join list
passed to it which, if not copied by the original caller, can introduce
bugs in the calling code if the original columns_to_join list is re-used
later, e.g. making nova objects with an expected_attrs list that has
been mutated by the DB API call.

This change makes a copy of the original list and passes back the new
list for the DB API methods to use, and adds some documentation to
hopefully clarify intent and usage.

A test is added to demonstrate the expected behavior of the utility
method.

Closes-Bug: #1393491

Change-Id: I49511f608291b66642ced00a3c13833e18fab3d7
Set vm state error when raising unexpected exception in live migrate

The instance stuck at migrating task_state when unexpected exception.
This is confuse for user, this patch set the vm_state to error.

Change-Id: Ib1b97452bc5e777c66c4d368f71156dbe1e116b7
Partial-Bug: 1397153
Add shelve and unshelve info into devref doc

The diagram on http://docs.openstack.org/developer/nova/devref/vmstates.html
does not currently show the "SHELVED_OFFLOADED" state and the
associated "shelve", "shelveOffload" and "unshelve" state transitions.

Change-Id: I50eb07a84336160b8120ca294598576eeb982d28
Closes-Bug: #1394215
Libvirt: SMB volume driver

Currently, there are Libvirt volume drivers that support
network-attached file systems such as Gluster or NFS. This patch
adds a new volume driver in order to support attaching volumes
hosted on SMB shares.

Co-Authored-By: Gabriel Samfira <gsamfira@cloudbasesolutions.com>

DocImpact

Change-Id: I1db3d2a6d8ee94932348c63cc03698fdefff0b5c
Implements: blueprint libvirt-smbfs-volume-support
Compute: Catch binding failed exception while init host

While compute starts it will init all instances,
if an exception is raised from one instance
(e.g NovaException during plug_vifs), then the
compute process exits unexpectedly because of
this unhandled exception.
This commit changes the NovaException to more
appropriate VirtualInterfacePlugException and
catches it during init host, as well as the
instance is set to error state, with this change
the compute process can be started normally even
if this VirtualInterfacePlugException is raised.

Closes-bug: #1324041

Change-Id: Ia584dba66affb86787e3069df19bd17b89cb5c49
Use a copy of NEW_NETWORK for test_networks

In test_networks, NEW_NETWORK is used as a request body between unit
tests. "Create a network" API changes a request body internally, so
each test should use a copy of NEW_NETWORK to avoid affecting to the
other unit tests. However one test uses NEW_NETWORK directly.
This patch makes each test use a copy of NEW_NETWORK for avoiding
similar problem in the future.

Change-Id: I9e12fe89956d18dde54bec83c15af258d66e0682
Add a validation format "cidr"

Nova accepts CIDR attributes on network REST APIs and checks its format
on each API. On v2.1 API, the attributes will be checked with JSONSchema.
In addition, Neutron contains its own "cidr" check for its REST APIs.
This patch add a validation format "cidr" by porting Neutron's check from
https://github.com/openstack/neutron/blob/master/neutron/api/v2/attributes.py#L293

Partially implements blueprint v2-on-v3-api

Change-Id: I03601f8fa34fe7617e4f963cdfd4b178eae36565
Share server access ips tests between V2 & V2.1

As server access ips should behave same in V2 and V2.1,
access ips tests can be shared among V2 & V2.1 to cleanup
duplicated test code.

This patch share those tests between V2 & V2.1

Partially implements blueprint v2-on-v3-api

Change-Id: Ife0b3f35ec6bd0d3c0052d2a8c0f314638a07ca3
Check for floating IP pool in nova-network

There are no checks in nova-network to see if a specified
floating IP pool exists or not before attempting to allocate
an IP address. If a pool does not exist, nova throws an
unrelated error - 'No more floating ips available'. The
following patch adds a check for the existence of the IP pool
and throws a FloatingIpPoolNotFound if it is not found.

Change-Id: I72cb8dd044f7ac094287bad7df9167b8e77a4b5b
Related-Bug: #1355623
downgrade 'No network configured!' to debug log level

Based on the details in bug 1191044, a user can cause the warning
'No network configured!' if they boot an instance in a tenant that
has no neutron networks set up.

Tempest doesn't create a network for every test that creates a nova
instance. The lack of a network is not a sign of something wrong with
nova so a warning log is too much. Move to debug instead.

This should help reduce the amount of warnings that occur on every
tempest run.

Also drop the exclamation mark.

Change-Id: I6bc4e46aa5a72645c72f7232627c7aa9efd2591d
Related-Bug: #1191044
Parse "networks" attribute if loading os-networks

"networks" attribute of "create a server" API should be used when
Neutron/os-network extension is loaded as TODO which is removed with
this patch mentions. V2.1 os-network extension has been implemented
with I94b7b476cbb02725a396725a379417b64afd58a7 already, so this patch
enables "networks" attribute for os-network extension.

Partially implements blueprint v2-on-v3-api

Change-Id: Ibe59ddc53d074f5b8147e24b35066d38d73c7c33
Adds hacking check for api_version decorator

Because of the implementation of this decorator and the controller's metaclass,
the api_version decorator must be the outermost (ie first) decorator wherever
it is used. This patch adds a hacking check to ensure that this is the case.

This decorator is intended to be used on multiple methods with the same name
which offends F811, so '#noqa' is needed too. This will be fixed in a separate
patch.

Bad:
    @some_decorator # noqa  <-- '# noqa' to avoid F811
    @api_version("2.5")     <-- Error, needs to be the first decorator
    def my_api_method...

Good:
    @api_version("2.5") # noqa   <-- this line still needs '# noqa' for F811
    @some_decorator
    def my_api_method...

Change-Id: I579c0061f03d788c477c5424d4d00ec7a6e721e1
Fix invalid read_deleted value in _validate_unique_server_name()

read_deleted passed to model_query() may have the value 'yes', 'no',
or 'only', but was being passed the value False. The implementation of
model_query caused this to be ignored, and the value of
context.read_deleted used instead.

This change cleans up kwarg parsing in model_query to ensure that all
values explicitly passed in retain their given value, unless they were
explicitly set to None, in which case they are defaulted anyway.  This
last is the expectation of current callers of model_query().

Changes Exception thrown for invalid values to ValueError.

Change-Id: I2d1bbdbd4a391ad3487da905df2e812166a837de
Add cn_get_all_by_host and cn_get_by_host_and_node to ComputeNode

Following the addition of the host field, now we can query by this field.
Here, there are additions for querying N nodes having same host and for
query 1 node having the tuple (host, node)

Here, we also need to provide a compatiblity layer in the ComputeNode
object to make sure that if old compute nodes are still running without
host field, we can still search against them.

Partially-Implements blueprint detach-service-from-computenode

Change-Id: Id23944060fd30da7293f02c31aa71c3150214794
Remove unused db.api.fixed_ip_get_by_address_detailed

Change-Id: I23ccad0460f38db5314180fda865153bd8784c01
libvirt: Fixes live migration for volume backed instances

Live migration fails for volume backed instances in LibvirtDriver
because _is_shared_block_storage() incorrectly identifies volume
backed disks as local. This is fixed by passing the block_device_info
parameter to get_instance_disk_info, which allows for correct
filtering of volume backed disks.

Change-Id: I1437b2a7d5a62615b0099114ed1b5b1110f56de2
Closes-bug: 1392773
Re-revert "libvirt: add version cap tied to gate CI testing"

In the following commit the 'version_cap' concept was
reverted in driver.py:

  commit 5dea55785a52b7edb383cf61fab2f65f2ae79a72
  Author: Sean Dague <sean@dague.net>
  Date:   Wed Jul 30 18:57:34 2014 +0000

    Revert "libvirt: add version cap tied to gate CI testing"

Unfortunately, due to a rebase error, when the version checking
code was moved into host.py, the version_cap was re-introduced:

  commit 069ad9a5b4b2cc9b49b21b09b0667909886961cf
  Author: Daniel P. Berrange <berrange@redhat.com>
  Date:   Fri Jul 4 15:12:38 2014 +0100

    libvirt: introduce new 'Host' class to manage the connection

This change reverts it once more

Change-Id: Icf4b2fff1c87d006c9827e5d27b1124e24255c0a
replace httplib.HTTPSConnection in EC2KeystoneAuth

httplib.HTTPSConnection is known to not verify SSL certificates
in Python 2.x. This change replaces use of httplib.HTTPSConnection
with the requests module. It imports config settings related to SSL
verification: ssl.key_file, ssl.cert_file, and ssl.ca_file. It also
adds one config setting: keystone_ec2_insecure. By default, SSL
verification is on, but can be disabled by setting:

keystone_ec2_insecure=true

This patch is based on the keystone middleware ec2 token patch:

https://review.openstack.org/#/c/76476

SecurityImpact
DocImpact
Closes-Bug: #1373992

Change-Id: I8e46d41164e9478b820cad569ba82f25de244620