xenapi: make boot from volume use volumeops

Currently the boot from volume code path has partially duplicated code
with the connect_volume method in volumeops and get_vdi_uuid_for_volume
in vm_utils.

This change removes the code duplication by calling volumeops to connect
the volume when doing boot from volume. For dependency reasons, this
means moving get_vdis_for_instance from vm_utils into vmops.

The unit test test_vdis_for_instance_bdi_password_scrubbed only checked
that auth_password was scrubbed if it was passed into the logging in a
certain way. The test now checks that auth_password is actually
passed into the logging in that way too.

Change-Id: I8591f30b177db38073a0145b902b607bb012a18a
xenapi: Attach original local disks during rescue

When rescuing an instance, a new VM is created and only the original
root disk is re-attached. Often when a user is rescuing a VM, they expect
to be able to access all of their original disks so they can potentially
salvage data.

This changes the xenapi driver to attach the original local disks
during rescue so the user can rescue all of their data.

DocImpact

Implements: blueprint rescue-attach-all-disks
Change-Id: Iba5cc85cd03d0a60f1858cf16aa31397e163df50
Partial-bug: 1223396
Fix nova boot failure using admin role for another tenant

When user create instances with admin role for another tenant
and specify a security group which name is duplicated in Neutron.
Then user will run into an error and from the GUI, and user can't
know what happened.

The root cause is the method find_resourceid_by_name_or_id doesn't
honoured the project info. So this fix will specify the project id
for find_resourceid_by_name_or_id(), so this patch depends on the
neutron client fix, see: https://review.openstack.org/#/c/103476/,
which has been merged.

Change-Id: I478ff4faa013a822483e39c8f5153a3a1e777f9b
Closes-Bug: 1335733
Use JSON instead of json in the parameter descriptions

The correct short form of "JavaScript Object Notation" is JSON.

Change-Id: I230d3abc5bb576694e06ecbf3e0be3324b28b5aa
VMware: add in DVS VXLAN support

The neutron network will be a VLAN port group or a VXLAN port group.
In the case of the latter the backend adds a VXLAX prefix to the
port group name.

The prefix is  'vxw-<dvs moref>-<virtual wiremoref>-sid-'

The patch supports both types of networks and enables VM's to have
network connectivity.

Change-Id: Icb93585eb486d4b6c8f141fd133971e2b0968f92
Work on document structure and doc building

This patch clarifies the document structure of
doc/source/devref/development.environment.rst and adds a section about
how to build the documentation.  This patch also strips out the
redundant information in doc/README.rst, replacing it with a reference
to doc/source/devref/development.environment.rst.

Change-Id: Ia354dcbaea35b812327dac7371ea2fe9aec2b594
Closes-Bug: 1268424
Fix extension parameters in test_multiple_create

In create() method of server, nova API parses extension parameters
from each extension's server_create() method by the following call:

   self.create_extension_manager.map(self._create_extension_point,
                                     server_dict, create_kwargs)

block_device_mapping value is parsed from 'os-block-device-mapping:
block_device_mapping' in os-block-device-mapping extension, and
min_count value is done from 'os-multiple-create:min_count'.
This patch fixes some wrong extension parameters in unit tests.

os-block-device-mapping extension calls BlockDeviceDict.from_api()
method for getting bdm_type and uuid from "source_type" and "uuid",
but current these parameters also are wrong.
This patch fixes test block_device_mapping values also.

These bugs were found through jsonschema validation framework
development, because the framework doesn't allow undefined
parameters based on each schema.

Change-Id: I0d5899c84c7a7f2938a553d7ce845f855f1dfd7a
Create a Scheduler client library

In order to split the scheduler code, a new library needs to be
created as a client for other Nova components.
Direct calls to conductor for updating stats in ResourceTracker
need to be decoupled into this library.

Change-Id: Ifcfeb90e09a75fc25ac5fd4f18cfd69ea0b21f50
Implements: blueprint scheduler-lib
Add support for select_destinations in Scheduler client

It was defined in the spec that the scheduler will provide a clear
interface for all scheduler API. Select_destinations() so needs
to be added in the client library.

Implements blueprint scheduler-lib

Change-Id: Ie5732baf9709cd0cb951eae4638910372c79e5f1
Prepend /dev/ to root_device_name in get_next_device_name

Because the API docs for booting an instance from a volume have shown to
use 'vda' for a long time, and it has worked, there may be instances
with root_device_name = 'vda'.  These instances will fail a check in
get_next_device_name when attempting to attach another volume, making
that impossible.  Another patch fixes new block_device_mappings by
prepending /dev/ when the root_device_name is populated, but this will
allow incorrectly populated instances to attach volumes.

Change-Id: I709a195c7e3f559315a2c307de62cbae8b72ac46
Partial-Bug: #1337821
VMware: spawn refactor add VirtualMachineConfigInfo

Class to capture parameters need to build and configure
a new virtual machine instance during spawn.

partial blueprint: vmware-spawn-refactor
Change-Id: I5ed435718cd78efe01c75b9b2aac6a13da405fc2
Stop augmenting oslo-incubators default log levels

Thanks to I255a68fc60963386e8fefe65c3ffd269795adbf4, the two modules we
explicitly set log levels for are now in oslo-incubator, so we don't
need the extra code introduced  in
Ib5164a22c5cbfa4f9d881b97bb1f623cd4d2f3f3

Change-Id: Iec7bd35da36135fcaf63a384fa35f849a1319738
Port os-server-groups extension to work in v2.1/v3 framework

Ports v2 os-server-groups extension and adapts it to the v2.1/v3 API
framework. API behaviour is identical with the exception
that there is no support for XML. Also

- unittest code modified to share testing with both v2/v2.1
- Adds expected error decorators for API methods
- Adds api sample tests

Partially implements blueprint v2-on-v3-api

Change-Id: I52e49e6e85a7516e3b184601374e5c987f60a9a0
Convert deferred_delete v3 plugin to v2.1 API

Changes required to have v3 plugin natively support the V2.1
API

The force_delete action is reverted back to forceDelete

Partially implements blueprint v2-on-v3-api

Change-Id: I0b050d3328338923cf05521b9ef5cc3b1c50a586
Change v3 os-user-data extension to v2.1

This patch changes v3 os-user-data extension to v2.1 and makes v2 unit
tests share between v2 and v2.1.

The differences between v2 and v3 are described on the wiki page
https://wiki.openstack.org/wiki/NovaAPIv2tov3 .

Partially implements blueprint v2-on-v3-api

Change-Id: Ia45398c2ec7ff40851f94260dc10c53766581ff4
Change v3 flavors to v2.1

This patch changes v3 flavors API to v2.1 and makes v2
unit tests share between v2 and v2.1.

The differences between v2 and v3 are described on the wiki page
https://wiki.openstack.org/wiki/NovaAPIv2tov3.

Partially implements blueprint v2-on-v3-api

Change-Id: I1e52c034a8c50fde3aa31e1e79c8f113424fd2e1
Change v3 flavor_rxtx to v2.1

This patch changes v3 flavor_rxtx API to v2.1 and makes v2
unit tests share between v2 and v2.1.

The differences between v2 and v3 are described on the wiki page
https://wiki.openstack.org/wiki/NovaAPIv2tov3.

Partially implements blueprint v2-on-v3-api

Change-Id: I0b79be078ff2b2ca682f1f061f40c81d4487c7de
Make test_killed_worker_recover faster

I3f7bd0ba5ef1bb716a97635252abe251053a669d changed how server.wait()
works so that it wouldn't return until all children are done. But
test_killed_worker_recover doesn't try to kill the children, unlike other
tests in test_multiprocess_api. This didn't cause the unit test to fail
because tearDown catches the unit test timeout, fixtures.TimeoutException,
which triggers after 160 seconds.

Now instead of taking 160 seconds test_killed_worker_recover, which is
run twice, takes under 3 seconds.

Change-Id: I8cbd08cb873cd81a65005764759ac9ca23fa6430
Closes-Bug: #1360719
Change v3 hosts to v2.1

This patch changes v3 hosts API to v2.1 and makes v2
unit tests share between v2 and v2.1.

The differences between v2 and v3 are described on the wiki page
https://wiki.openstack.org/wiki/NovaAPIv2tov3.

Partially implements blueprint v2-on-v3-api

Change-Id: I8ce66570edb06bde0bdea677e6e784539aaab763
Update instance state after compute service died for rebuilded instance

When nova-compute died unexpectedly or rpc message lost unexpectly
and an rebuild operation is ongoing, the instance will be in REBUILD
state forever after the nova-compute service restarts.
Only if admin take actions such as reset its state.
Like what we did for build instance (general spawn)
operation, the instance might not be used so it need to be set to ERROR
state after compute service restarts.

Change-Id: I37748edb3ca0501b6f2fb41fbd75b7b8aa00d59c
Closes-Bug: #1304184