xenapi: make boot from volume use volumeops

Currently the boot from volume code path has partially duplicated code
with the connect_volume method in volumeops and get_vdi_uuid_for_volume
in vm_utils.

This change removes the code duplication by calling volumeops to connect
the volume when doing boot from volume. For dependency reasons, this
means moving get_vdis_for_instance from vm_utils into vmops.

The unit test test_vdis_for_instance_bdi_password_scrubbed only checked
that auth_password was scrubbed if it was passed into the logging in a
certain way. The test now checks that auth_password is actually
passed into the logging in that way too.

Change-Id: I8591f30b177db38073a0145b902b607bb012a18a
Don't truncate osapi_glance_link or osapi_compute_link prefixes

When osapi_glance_link_prefix or osapi_compute_link_prefix is
defined and used to update the links return in API responses, do
not drop the path component of the overriding link prefix.

Change-Id: I290e3f85b94e28d25a54621c1699081cbbf48456
Closes-Bug: 1304099
Implement methods to modify volume metadata.

There are four methods in nova/volume/cinder.py which are
NotImplemented. They are as follows:
  1. get_volume_metadata
  2. delete_volume_metadata
  3. update_volume_metadata
  4. get_volume_metadata_value
These methods are required in cases where nova needs to modify
a cinder volume's metadata, e.g. attach and detach time. It also
completes the cinder.API class.

Change-Id: I360076ae28db43e661466f556425947813c9040e
Closes-Bug: #1296164
Keep Migration status in automatic confirm-resize

function in compute/manager.py _poll_unconfirmed_resizes
will translate the migrate status from finished to error
whenever finds a problem ,Consider the following case

1) _poll_unconfirmed_resizes runs and founds several migrations to be confirmed
2) user deletes the instance, so its task state will be changed to DELETING
3) _poll_unconfirmed_resizes will find the task state is not None,
it will make the migration error status
4) following code in _delete
if instance.vm_state == vm_states.RESIZED:
self._confirm_resize_on_deleting(context, instance)
will fail because migration status already updated

User is able to do the migration operations when they found error
so we don't need to handle this kind of error for them

Change-Id: Ia79dd56c9c1fa8add7dc138d838cbfbac4523ac3
Closes-Bug: #1301696
Handle a flavor create failed better

Catch FlavorCreateFailed exceptions which when flavor creation fails.
This type of flavor creation failure is caused by an internal nova problem.
Return an internal server error (500) to the user so they know
to contact their cloud provider.

Closes-Bug: #1292309

Change-Id: I268c84dae290b0170448d789509af781555df9b0
libvirt: VM diagnostics (v3 API only)

There is no formal definition for the VM diagnostics. For the
V2 API the diagnostics will be returned as they are today.

This will support backward compatibility with the existing API's.

Part of the blueprint v3-diagnostics

DocImpact

Change-Id: I4128e76622e73a40d25fb50b10b1c1ac710a5cc4
xenapi: Attach original local disks during rescue

When rescuing an instance, a new VM is created and only the original
root disk is re-attached. Often when a user is rescuing a VM, they expect
to be able to access all of their original disks so they can potentially
salvage data.

This changes the xenapi driver to attach the original local disks
during rescue so the user can rescue all of their data.

DocImpact

Implements: blueprint rescue-attach-all-disks
Change-Id: Iba5cc85cd03d0a60f1858cf16aa31397e163df50
Partial-bug: 1223396
VMWare Driver - Ignore datastore in maintenance mode

A datastore can be part of a datastore cluster. Any
datastore which is part of a cluster can be put in maintenance
mode. Instances cannot reside on a datastore if it is in
maintenance mode.
This fix ignores a datastore in the following scenarios
1. Stats update - Ignore a datastore during stats update.
2. Ignore a datastore during instance spawn.

Change-Id: Id0cb3f5fdb4d0bc4884cf5405fadc433d4a2b6ba
Closes-Bug: #1330065
Fix fake_update in test_update_missing_server

In test_update_missing_server, fake_update should be used as update()
method but it is used as create() instead. In addition, the test comment
also is wrong.
This patch fixes them.

Change-Id: Ia4704413dd7355010546dba7096ed34225bb4b0f
Merge BadRequest tests of "attach interfaces" API

In test_attach_interfaces, there are BadRequest tests of "attach interfaces".
Most parts of them are duplicated.
This patch merges them for the readability and clarifying their purposes.

Change-Id: Ib9dd2840be46dec6b70fe9e3fda35de88f84b4a9
Fix nova boot failure using admin role for another tenant

When user create instances with admin role for another tenant
and specify a security group which name is duplicated in Neutron.
Then user will run into an error and from the GUI, and user can't
know what happened.

The root cause is the method find_resourceid_by_name_or_id doesn't
honoured the project info. So this fix will specify the project id
for find_resourceid_by_name_or_id(), so this patch depends on the
neutron client fix, see: https://review.openstack.org/#/c/103476/,
which has been merged.

Change-Id: I478ff4faa013a822483e39c8f5153a3a1e777f9b
Closes-Bug: 1335733
Use JSON instead of json in the parameter descriptions

The correct short form of "JavaScript Object Notation" is JSON.

Change-Id: I230d3abc5bb576694e06ecbf3e0be3324b28b5aa
Network: add in a new network type - DVS

The support enables DVS network backends. In this case neutron
will return a vif type of DVS.

The name of the VC port group will be of the following format:
    <network-name>-<network-uuid>

The network name is so that the prefix will be easy for people to
understand. The UUID is to ensure that it will be unique.

Change-Id: Id125797632591bd799e215da7cbae07154e90344
VMware: add in DVS VXLAN support

The neutron network will be a VLAN port group or a VXLAN port group.
In the case of the latter the backend adds a VXLAX prefix to the
port group name.

The prefix is  'vxw-<dvs moref>-<virtual wiremoref>-sid-'

The patch supports both types of networks and enables VM's to have
network connectivity.

Change-Id: Icb93585eb486d4b6c8f141fd133971e2b0968f92
VMware: Do not read opaque type for DVS network

Save backend calls when using DVS VIF type. There is no need to
get the opaque network in the event that we have a DVS network.

Change-Id: I75973005193f862dfde3b99167cb53d3114a8349
Adds verbosity to child cell update log messages

Child Cells frequently update their parents with their capabilities and
capacities. The debug log message contains the capabilities or capacities
that are being updated. This patch adds the names of the parent cells to
which the information is being sent to the debug log message.

Change-Id: Ic34191da191369e64a016c8300a8da294aaa9e07
Closes-Bug: 1351301
Remove pre-icehouse rpc client API compat

The compute rpc API was at version 3.23 for the Icehouse release.
Since we only support live upgrades from N-1 to N (Icehouse to Juno),
we can remove old rpc client support for versions that were
pre-Icehouse release.  This patch does that for the compute rpcapi.

Change-Id: Ice20b141dbe28de02e125fa5ced1dbc286ba2ee3
Work on document structure and doc building

This patch clarifies the document structure of
doc/source/devref/development.environment.rst and adds a section about
how to build the documentation.  This patch also strips out the
redundant information in doc/README.rst, replacing it with a reference
to doc/source/devref/development.environment.rst.

Change-Id: Ia354dcbaea35b812327dac7371ea2fe9aec2b594
Closes-Bug: 1268424
Fix extension parameters in test_multiple_create

In create() method of server, nova API parses extension parameters
from each extension's server_create() method by the following call:

   self.create_extension_manager.map(self._create_extension_point,
                                     server_dict, create_kwargs)

block_device_mapping value is parsed from 'os-block-device-mapping:
block_device_mapping' in os-block-device-mapping extension, and
min_count value is done from 'os-multiple-create:min_count'.
This patch fixes some wrong extension parameters in unit tests.

os-block-device-mapping extension calls BlockDeviceDict.from_api()
method for getting bdm_type and uuid from "source_type" and "uuid",
but current these parameters also are wrong.
This patch fixes test block_device_mapping values also.

These bugs were found through jsonschema validation framework
development, because the framework doesn't allow undefined
parameters based on each schema.

Change-Id: I0d5899c84c7a7f2938a553d7ce845f855f1dfd7a
add log exception hints in some modules

Add hints for log exception messages

Change-Id: I51b30d58bcb4b9fa8e51b29af1cfff33c0b432aa