xenapi: make boot from volume use volumeops

Currently the boot from volume code path has partially duplicated code
with the connect_volume method in volumeops and get_vdi_uuid_for_volume
in vm_utils.

This change removes the code duplication by calling volumeops to connect
the volume when doing boot from volume. For dependency reasons, this
means moving get_vdis_for_instance from vm_utils into vmops.

The unit test test_vdis_for_instance_bdi_password_scrubbed only checked
that auth_password was scrubbed if it was passed into the logging in a
certain way. The test now checks that auth_password is actually
passed into the logging in that way too.

Change-Id: I8591f30b177db38073a0145b902b607bb012a18a
Fix nova boot failure using admin role for another tenant

When user create instances with admin role for another tenant
and specify a security group which name is duplicated in Neutron.
Then user will run into an error and from the GUI, and user can't
know what happened.

The root cause is the method find_resourceid_by_name_or_id doesn't
honoured the project info. So this fix will specify the project id
for find_resourceid_by_name_or_id(), so this patch depends on the
neutron client fix, see: https://review.openstack.org/#/c/103476/,
which has been merged.

Change-Id: I478ff4faa013a822483e39c8f5153a3a1e777f9b
Closes-Bug: 1335733
Use JSON instead of json in the parameter descriptions

The correct short form of "JavaScript Object Notation" is JSON.

Change-Id: I230d3abc5bb576694e06ecbf3e0be3324b28b5aa
VMware: add in DVS VXLAN support

The neutron network will be a VLAN port group or a VXLAN port group.
In the case of the latter the backend adds a VXLAX prefix to the
port group name.

The prefix is  'vxw-<dvs moref>-<virtual wiremoref>-sid-'

The patch supports both types of networks and enables VM's to have
network connectivity.

Change-Id: Icb93585eb486d4b6c8f141fd133971e2b0968f92
Create a Scheduler client library

In order to split the scheduler code, a new library needs to be
created as a client for other Nova components.
Direct calls to conductor for updating stats in ResourceTracker
need to be decoupled into this library.

Change-Id: Ifcfeb90e09a75fc25ac5fd4f18cfd69ea0b21f50
Implements: blueprint scheduler-lib
Add support for select_destinations in Scheduler client

It was defined in the spec that the scheduler will provide a clear
interface for all scheduler API. Select_destinations() so needs
to be added in the client library.

Implements blueprint scheduler-lib

Change-Id: Ie5732baf9709cd0cb951eae4638910372c79e5f1
Prepend /dev/ to root_device_name in get_next_device_name

Because the API docs for booting an instance from a volume have shown to
use 'vda' for a long time, and it has worked, there may be instances
with root_device_name = 'vda'.  These instances will fail a check in
get_next_device_name when attempting to attach another volume, making
that impossible.  Another patch fixes new block_device_mappings by
prepending /dev/ when the root_device_name is populated, but this will
allow incorrectly populated instances to attach volumes.

Change-Id: I709a195c7e3f559315a2c307de62cbae8b72ac46
Partial-Bug: #1337821
VMware: spawn refactor add VirtualMachineConfigInfo

Class to capture parameters need to build and configure
a new virtual machine instance during spawn.

partial blueprint: vmware-spawn-refactor
Change-Id: I5ed435718cd78efe01c75b9b2aac6a13da405fc2
Change v3 os-user-data extension to v2.1

This patch changes v3 os-user-data extension to v2.1 and makes v2 unit
tests share between v2 and v2.1.

The differences between v2 and v3 are described on the wiki page
https://wiki.openstack.org/wiki/NovaAPIv2tov3 .

Partially implements blueprint v2-on-v3-api

Change-Id: Ia45398c2ec7ff40851f94260dc10c53766581ff4
Change v3 flavor_rxtx to v2.1

This patch changes v3 flavor_rxtx API to v2.1 and makes v2
unit tests share between v2 and v2.1.

The differences between v2 and v3 are described on the wiki page
https://wiki.openstack.org/wiki/NovaAPIv2tov3.

Partially implements blueprint v2-on-v3-api

Change-Id: I0b79be078ff2b2ca682f1f061f40c81d4487c7de
Make test_killed_worker_recover faster

I3f7bd0ba5ef1bb716a97635252abe251053a669d changed how server.wait()
works so that it wouldn't return until all children are done. But
test_killed_worker_recover doesn't try to kill the children, unlike other
tests in test_multiprocess_api. This didn't cause the unit test to fail
because tearDown catches the unit test timeout, fixtures.TimeoutException,
which triggers after 160 seconds.

Now instead of taking 160 seconds test_killed_worker_recover, which is
run twice, takes under 3 seconds.

Change-Id: I8cbd08cb873cd81a65005764759ac9ca23fa6430
Closes-Bug: #1360719
Change v3 hosts to v2.1

This patch changes v3 hosts API to v2.1 and makes v2
unit tests share between v2 and v2.1.

The differences between v2 and v3 are described on the wiki page
https://wiki.openstack.org/wiki/NovaAPIv2tov3.

Partially implements blueprint v2-on-v3-api

Change-Id: I8ce66570edb06bde0bdea677e6e784539aaab763
Update instance state after compute service died for rebuilded instance

When nova-compute died unexpectedly or rpc message lost unexpectly
and an rebuild operation is ongoing, the instance will be in REBUILD
state forever after the nova-compute service restarts.
Only if admin take actions such as reset its state.
Like what we did for build instance (general spawn)
operation, the instance might not be used so it need to be set to ERROR
state after compute service restarts.

Change-Id: I37748edb3ca0501b6f2fb41fbd75b7b8aa00d59c
Closes-Bug: #1304184
Use default quota values in test_quotas

There are much copy&paste code in test_quotas of v2 API.
This patch adds default quota values and makes use it for each API test
for the readability.

Change-Id: Ib40a1820ea0db93daf231b3384f203ab0ef696ba
Merge BadRequest tests of v2 quota_sets API

In test_quotas, there are BadRequest tests of v2 quota_sets API.
Most parts of them are duplicated.
This patch merges them for the readability and clarifying their purposes.

Change-Id: I4313b2a80819260e9363f457ba271689308b145f
Convert v3 server diagnostics plugin to v2.1

This patch changes v3 server diagnostics plugin to v2.1 and makes v2
unit tests share between v2 and v2.1.

This patch also delete the V3 server_diagnostics API tests.

The differences between v2 and v3 are described on the wiki page
https://wiki.openstack.org/wiki/NovaAPIv2tov3.

Partially implements blueprint v2-on-v3-api

Change-Id: If10615f3f119dda815d56cf8329619bab502e091
Explicitly handle exception ConsoleTypeUnavailable for v2 consoles

In v3 remote_console extension, the exception ConsoleTypeUnavailable
is catched by explicitly. This patch changes the v2 api as same way,
and backport v3 unittest into v2 api.

Change-Id: I14f74e4a4b4ee84713b48cfc00d3f132726f9316
Handle non dict metadata in server metadata V2 API

While creating or updating the server metadata, requested metadata
should be in form of dictionary.
Currently if it is not in that form then, 500 error is being return
which should be 400.

This patch adds such request check for V2 API and adds its UT

Change-Id: Ifcd4f9d352de7b2676beff319b641547b8952566
Updates available resources after live migration

A new VM can't be deployed using resources of the VM that has been migrated
to a new host before the periodic task is called after a live migration.

This change calls update_available_resource() after live
migration is completed even before the next periodic task takes place.

Change-Id: I4c879dcfb1e76cfdc95612b6d8010c1081ac45b9
Closes-Bug: #1267862
Plumb NetworkRequest objects through conductor and compute RPC

This makes the conductor and compute RPC paths for building instances
support and use NetworkRequest objects, which gets this all the way
down to the compute managers.

Change-Id: Ia45e15896b80ae33533d77719dd23bbdf60a1945