xenapi: virt rescue method now supports objects

Commit a37a96fd13b933123e80ad24e37fc37d73c16059 ensured
that the compute API and manager are now getting objects
for the rescue method.

This patch adds support to the virt driver code.

Change-Id: I06e5032526f791dfc0c523f2302d2edff5023f78
xenapi: virt unrescue method now supports objects

Commit 65ad230af5ece020822fa963dd30fd641a54ae91 ensured
that the compute API and manager are now getting objects
for the unrescue method.

This patch adds support to the virt driver code.

Change-Id: Ie6b587d3f89202a6987291bfaea88290241651f2
Use VIF details dictionary to get physical_network

Modify mlnx_direct plug and unplug to retrieve physical_network
from VIF details dictionary if network meta dictionary does not contain
physical_network. This will serve ML2 case following the guidelines to
populate vif_details dictionary with attributes required for port plugging.

Change-Id: I0d97bc875be2fff18087c78accef3ec81c059c4b
Closes-Bug: 1304872
Fixes hyper-v volume attach when host is AD member

The domain name gets added to the initiator name used by the host
if it's an Active Directory member.

Currently the method which gets the initiator name does not take
this into account when retrieving the default initiator name.

Trying to use a wrong initiator name will lead to an exception when
trying to log in to the according iSCSI target.

This patch simply appends the domain name (when the host is an AD
member) to the initiator name used to log in to iSCSI targets.

Closes-Bug: #1328870

Change-Id: Ifbe762c685e46081059a01043431b2c4ac5473fc
Replace assertTrue(not *) with assertFalse(*)

Replacements were done in tests to have
clearer messages in case of failure

Change-Id: I7eedc62e0d570ee5818c4aa9b28ad8163c63b775
Add supported file system type check at virt layer

Currently nova doesn't check whether the file system type is supported
by virt layer(hypervisor) before use it. This patch adds current
support file system type definition then create a default check
function at virt layer. Following patches will do the real check
in different driver such as libvirt.

Change-Id: Ie4d876a48b36c1a53b171dd521bdeef868a31486
Partial-Bug: #1293880
Replace assertTrue/False with assertIn/NotIn

The following replacements were done in tests to have
clearer messages in case of failure:
- assertTrue(* in *) with assertIn
- assertTrue(* not in *) with assertNotIn
- assertFalse(* in *) with assertNotIn

assertTrue/False with custom error messages weren't
replaced because assertIn/NotIn doesn't have argument
'message' and we'll lose error messages if we replace it.

Also assertTrue/False with operator 'any' were
replaced with more simpler asserts assertIn/NotIn.

Change-Id: If5e78db21d5302259a4d69fd82c62baf39a5d8f8
Add API schema for v2.1/v3 hosts API

By defining the API schema, it is possible to separate the validation
code from the API method. The API method can be more simple.
In addition, a response of API validation error can be consistent for
whole of Nova APIs.

Partially implements blueprint v3-api-schema

Change-Id: Ie252dce67f8e9d3292d8dad767b6113f7affcffd
Add API schema for v2.1/v3 flavors_extraspecs API

By defining the API schema, it is possible to separate the validation
code from the API method. The API method can be more simple.
In addition, a response of API validation error can be consistent for
the whole Nova API.

Partially implements blueprint v3-api-schema

Change-Id: I71f89b59dd4e542d2befa971182cc587b6fec269
Move rebuild to conductor and add find host logic

[*] Move rebuild instance to conductor task api will help
    in consolidating the logic for both evacuate instance and
    rebuild instance. Rebuild instance and evacuate instance
    uses the conductor task api for rebuild instance workflow.
[*] Add logic required to support rebuild without host.
    (used by evacuate)
[*] Refactor rebuild action to pass the instance host as argument,
    instead of None.

Evacuate and Rebuild will behave the same way as today.
Modifications for the new feature in API will be implemented in
new patches.

Partial Implements: blueprint find-host-and-evacuate-instance

Change-Id: I4b1a9c2227574db7179441d531a49145f213ea87
Co-Authored-By: Juan M. Olle <juan.m.olle@intel.com>
Co-Authored-By: Anuj Mathur <anujm@thoughtworks.com>
Co-Authored-By: Navneet Kumar <navneetk@thoughtworks.com>
Co-Authored-By: Claxton Correya <claxton@gmail.com>
Add APIv3 support to make host optional on evacuate.

DocImpact: The evacuate target host is now optional. If
'host' field is not sent in the request, the scheduler
will determine the target host. This will include
nova client changes ( on the proper commit ) to support
this new optional parameter.

Implements: blueprint find-host-and-evacuate-instance

Change-Id: I907ea4ef39cfad50a48cb80bceba217fdf482f6e
Co-Authored-By: Juan M. Olle <juan.m.olle@intel.com>
Co-Authored-By: Andres Buraschi <andres.buraschi@intel.com>
Co-Authored-By: Anuj Mathur <anujm@thoughtworks.com>
Co-Authored-By: Navneet Kumar <navneetk@thoughtworks.com>
Co-Authored-By: Claxton Correya <claxton@gmail.com>
Update port binding when unshelve instance

When unshelve instance, it should update the attached port also.
This patch invoke network_api.migrate_instance_finish to update
the port's binding info.

Change-Id: Iaa77163dde5494154d67ef392142669d70c3e643
Closes-Bug: #1306922
check ephemeral disk format at libvirt before use

Add ephemeral disk format at libvirt. There is no check before
so it will be used directly. This patch adds valid check function
and validate it before eph disk was used in virt layer.

Change-Id: I1934573e79f126e725cef4848b15c11a595161c9
Closes-Bug: #1293880
Mask node.session.auth.password in volume.py _run_iscsiadm debug logs

The iscsi_command object passed to _run_iscsiadm can contain passwords
that get logged at debug level, so we need to sanitize the message
getting logged.

Adds a test to ensure the logged message is properly sanitized.

Closes-Bug: #1320028

Change-Id: I33f1a5b698368504721b41e56266162a713b3ce6
libvirt: Save device_path in connection_info when booting from volume

If you boot an instance from a volume and later terminate it, the
libvirt volume driver disconnect_volume method does not have the
'device_path' key in connection_info['data'].  However, if you
attach a volume to an existing instance and then detach it,
the disconnect_volume method would have the 'device_path' key in
connection_info['data'].  Having the 'device_path' key would be
useful for some volume drivers to determine the device path of the
volume.  This patch saves the 'device_path' in connection_info['data']
when _create_domain_and_network is called, so it could be later used.

Change-Id: I8ebb5f3c2e7a81b11d776f8c0a15f3491ed273be
Closes-Bug: #1291007
Change compute updates from periodic to on demand

Currently, all compute nodes update status info in the DB on a periodic
basis (the period is currently 60 seconds). Given that the status of
the node only changes at specific points (mainly image
creation/destruction) this leads to significant DB overhead on a large
system. This BP changes the update mechanism to only update the DB when
a node state changes by keeping an old copy of the state info and
only updating the DB if the new state info is different.

Docimpact - check to see if there's a documentation impact to this change.

Change-Id: I19c174488b61977358a4d617cc4b590e317e665a
Implements: blueprint on-demand-compute-update
vmware: VM diagnostics (v3 API only)

There is no formal definition for the VM diagnostics. For the
V2 API the diagnostics will be returned as they are today.

This will support backward compatibility with the existing API's.

Part of the blueprint v3-diagnostics

DocImpact

Change-Id: I6465ee6fc2edac0107e260297d9647c8a16b09ae
VMware: validate the network_info is defined

Ensure that the network info is defined correctly prior to allocating
networks.

Commit 6fb9cc8b278ceab05e65e1c5145141203211d246 caused a degradation.

Change-Id: I74eddc0ff744ee5d80df825a52527c67583bacea
Move retry of prep_resize to conductor instead of scheduler

Prep_resize in Compute Manager calls back Scheduler prep_resize if
an Exception is raised. There is no sense of calling back the
scheduler, it should call Conductor instead.

Implements blueprint move-prep-resize-to-conductor

Change-Id: I2131843c291f5bd01eb4dbd6c8f36a9218120c19
XenAPI: VM diagnostics for v3 API

For V3 API the diagnostics will have the format defined at:
https://wiki.openstack.org/wiki/Nova_VM_Diagnostics

This will support backward compatability with the existing APIs

Part of the blueprint v3-diagnostics

DocImpact

Change-Id: I8377f86a97ca554028e4ca8c1c1af7fd5643cc2e