xenapi: Attach original local disks during rescue

When rescuing an instance, a new VM is created and only the original
root disk is re-attached. Often when a user is rescuing a VM, they expect
to be able to access all of their original disks so they can potentially
salvage data.

This changes the xenapi driver to attach the original local disks
during rescue so the user can rescue all of their data.

DocImpact

Implements: blueprint rescue-attach-all-disks
Change-Id: Iba5cc85cd03d0a60f1858cf16aa31397e163df50
Partial-bug: 1223396
Work on document structure and doc building

This patch clarifies the document structure of
doc/source/devref/development.environment.rst and adds a section about
how to build the documentation.  This patch also strips out the
redundant information in doc/README.rst, replacing it with a reference
to doc/source/devref/development.environment.rst.

Change-Id: Ia354dcbaea35b812327dac7371ea2fe9aec2b594
Closes-Bug: 1268424
Create a Scheduler client library

In order to split the scheduler code, a new library needs to be
created as a client for other Nova components.
Direct calls to conductor for updating stats in ResourceTracker
need to be decoupled into this library.

Change-Id: Ifcfeb90e09a75fc25ac5fd4f18cfd69ea0b21f50
Implements: blueprint scheduler-lib
Add support for select_destinations in Scheduler client

It was defined in the spec that the scheduler will provide a clear
interface for all scheduler API. Select_destinations() so needs
to be added in the client library.

Implements blueprint scheduler-lib

Change-Id: Ie5732baf9709cd0cb951eae4638910372c79e5f1
Prepend /dev/ to root_device_name in get_next_device_name

Because the API docs for booting an instance from a volume have shown to
use 'vda' for a long time, and it has worked, there may be instances
with root_device_name = 'vda'.  These instances will fail a check in
get_next_device_name when attempting to attach another volume, making
that impossible.  Another patch fixes new block_device_mappings by
prepending /dev/ when the root_device_name is populated, but this will
allow incorrectly populated instances to attach volumes.

Change-Id: I709a195c7e3f559315a2c307de62cbae8b72ac46
Partial-Bug: #1337821
Change v3 os-user-data extension to v2.1

This patch changes v3 os-user-data extension to v2.1 and makes v2 unit
tests share between v2 and v2.1.

The differences between v2 and v3 are described on the wiki page
https://wiki.openstack.org/wiki/NovaAPIv2tov3 .

Partially implements blueprint v2-on-v3-api

Change-Id: Ia45398c2ec7ff40851f94260dc10c53766581ff4
Change v3 flavors to v2.1

This patch changes v3 flavors API to v2.1 and makes v2
unit tests share between v2 and v2.1.

The differences between v2 and v3 are described on the wiki page
https://wiki.openstack.org/wiki/NovaAPIv2tov3.

Partially implements blueprint v2-on-v3-api

Change-Id: I1e52c034a8c50fde3aa31e1e79c8f113424fd2e1
Make test_killed_worker_recover faster

I3f7bd0ba5ef1bb716a97635252abe251053a669d changed how server.wait()
works so that it wouldn't return until all children are done. But
test_killed_worker_recover doesn't try to kill the children, unlike other
tests in test_multiprocess_api. This didn't cause the unit test to fail
because tearDown catches the unit test timeout, fixtures.TimeoutException,
which triggers after 160 seconds.

Now instead of taking 160 seconds test_killed_worker_recover, which is
run twice, takes under 3 seconds.

Change-Id: I8cbd08cb873cd81a65005764759ac9ca23fa6430
Closes-Bug: #1360719
Update instance state after compute service died for rebuilded instance

When nova-compute died unexpectedly or rpc message lost unexpectly
and an rebuild operation is ongoing, the instance will be in REBUILD
state forever after the nova-compute service restarts.
Only if admin take actions such as reset its state.
Like what we did for build instance (general spawn)
operation, the instance might not be used so it need to be set to ERROR
state after compute service restarts.

Change-Id: I37748edb3ca0501b6f2fb41fbd75b7b8aa00d59c
Closes-Bug: #1304184
Use default quota values in test_quotas

There are much copy&paste code in test_quotas of v2 API.
This patch adds default quota values and makes use it for each API test
for the readability.

Change-Id: Ib40a1820ea0db93daf231b3384f203ab0ef696ba
Convert v3 server diagnostics plugin to v2.1

This patch changes v3 server diagnostics plugin to v2.1 and makes v2
unit tests share between v2 and v2.1.

This patch also delete the V3 server_diagnostics API tests.

The differences between v2 and v3 are described on the wiki page
https://wiki.openstack.org/wiki/NovaAPIv2tov3.

Partially implements blueprint v2-on-v3-api

Change-Id: If10615f3f119dda815d56cf8329619bab502e091
GET servers API sorting enhancements common utilities

This change is to support updating the v2 and v3 /servers and
/servers/detail APIs to support multiple sort keys and sort directions
(using the 'sort_key' and 'sort_dir' parameters); these parameters can
be specified multiple times to create a list of sort keys and
directions.

This patch set contains utility functions to:
* Process the input sorting parameters on the request
* Process the sort keys and directions to include the default keys

The main complexity in this change deals with defaulting the sort keys
and directions so that server order does not change for existing
paths. A new 'process_sort_params' function is created to consistently
process the the sort parameters (ie, 'sort_keys', and 'sort_dirs')
that will be passed from the REST API, through the various layers, and
down to the DB API. This utility function will be invoked when the sort
params need to be processed (and defaulted) in the DB sqlalchemy layer
(prior to invoking the common paginate_query function).

These functions will be consumed in another change set.

Change-Id: I4253bc7d43142b783c14096d2834f53dcc0b7d3f
Partially Implements: blueprint nova-pagination
Fix error in log when log exception in guestfs.py

This code review: https://review.openstack.org/#/c/104262/
brings an issue when log exception in guestfs.py at
LOG.info(_LI("Unable to force TCG mode, libguestfs too old?")
To fix this issue we just need to add %s in it.

Change-Id: Ic33a6b7c2ce49e21e96a087984aba6656065d8fc
Closes-bug: #1352768
libvirt: reworks configdrive creation

This refactor the creation of the configdrive by using
the same code schema as local/ephemeral/swap/disk.

Now this is the imagebackend that creates the configdrive like any other
disk attached to VM. This ensures that the configdrive file is created
where the imagebackend expect it.

This also removes the assumption that the configdrive was always created
at the right place, that was not true for rbd and lvm.

Related-Bug: #1361840

Change-Id: Ia03a83d593794c8b2a644360790cb15de9dea6e9
Changes block_device_mapping extension into v2.1

This patch changes v3 block_device_mapping API to v2.1 and make
unittest share between v2 and v2.1

The differences between v2 and v3 are described on the wiki page
https://wiki.openstack.org/wiki/NovaAPIv2tov3.

Partially implements blueprint v2-on-v3-api

Change-Id: I935b336523241a32dbd2bb8b154ce72afefd3aa9
Catch BDM related InvalidBDM exceptions for server create v2.1

For v2.1 api, we should catch exception explicitly. There are some
BDM related InvalidBDM* exceptions missing catch. This patch fix this.
Also move the related unittests into block_device_mapping unittests.

Change-Id: Id1bef4adc71ffedd52eb9278b5d09954fc1b1193
Add extension block_device_mapping_v1 for v2.1

In the beginning v3 didn't support legacy block_device_mapping request.
For v2.1, we want to keep back-compatibility with v2 api, so we need
add legacy block_device_mapping.

This patch add new extension block_device_mapping_v1 for v2.1. Also
share legacy block_device_mapping related unittests between v2 and v2.1.

Partially implements blueprint v2-on-v3-api

Change-Id: I841833d8fd0995ddf49febd9c1611cf711742f5a
Convert v3 certificate API to v2.1

This patch changes v3 certificate API to v2.1 and makes v2
unit tests share between v2 and v2.1.

The differences between v2 and v3 are described on the wiki page
https://wiki.openstack.org/wiki/NovaAPIv2tov3.

Partially implements blueprint v2-on-v3-api

Change-Id: I8e6556dffbc14334f7ec1b9db38fbbfe2387bef0
Move and generalize decorator serialize_args to nova.objects.base

It would be nice use decorator `serialize_args` as general-use
decorator. Note that this single-purpose decorator was covering
up our ability to drill down to the actual method implementation
for hash calculation in the TestObjectVersions test. This patch
also enhances the decorator to be transparent, and makes the
test able to drill down through multiple layers of decorators
to find the actual implementation.

Co-Authored-By: Dan Smith <dansmith@redhat.com>
Change-Id: Ie454bffba113d7b3a863e975b2fb0ee6fb1dd552
Add instance_extra table and related objects

This patch adds the table for storing the instance NUMA topology, as
adding it on the model itself as a text field is thought to be bad for
perfomance.

We need to add it as scheduling based on it and tracking resources
proves to be impossible otherwise as we would have to rely on
re-calculating it from data stashed in system_metadata table, which is
deemed even more inefficient if it needs to be done in a periodic task.

We add the objects so that we can make sure that data is versioned, and
the relevant database models methods and the migration for adding the
table.

Blueprint: virt-driver-numa-placement
Change-Id: I724ed7089044177ec4fecdf4fdcefad96a2466b6