Make baremetal support metadata for ephemeral block-device-mapping

Make baremetal driver set instance['default_ephemeral_device'] to
/dev/sda1 since we are using the same device for root(sda3) and
ephemeral(sda1)

Change-Id: Ib2076198eacb0ea61706043b42d673c35d26f5d1
Partial-Bug: #1239524
Users with admin role in Nova should not re-auth with Neutron

A recent change to the way Nova creates a Neutron clients
https://review.openstack.org/#/c/52954/4 changed the conditions
under which it re-authenticates using the neutron admin credentials
from "if admin" to "if admin or context.is_admin".

This means that any user with admin role in Nova now interacts
with Neutron as a different tenant.  Not only does this cause an
unnecessary re-authentication (The user may/should also have an admin
role in Neutron) it means that they can no longer allocate and assign
a floating IP to their instance via Nova (as the floating ip will now
always be allocated in the context of neutron_admin_tenant).

This change removes the context.is_admin check to revert to the
previous behaviour where the used of admin creds is determined just
on the explicit use of "admin=True" parameter to get_client().

Change-Id: Ib1720420c778960bc90c5b7d703de936ebb7d6b5
Closes-Bug: 1250763
Added monitor (e.g. CPU) to monitor and collect data

This patch gives a monitor interface for users to add their own monitors to
monitor on compute nodes on the compute manager.

The compute manager loads the monitors and gets the metrics, and sends to
the scheduler for utilization based scheduling.
The patch also gives a sample to collect CPU utilization data by libvirt
driver. The patch brings a new API get_host_cpu_stats() in class
ComputeDriver, and libvirt has implemented that in the patch.
For other virt drivers, it has not been implemented yet (TODO).

This is part of the blueprint utilization-aware-scheduling and the
blueprint generic-host-state-for-scheduler.

Change-Id: I4cd4f972c5c1cf6242a77aaeb9514c8a0d4faa0d
Fix libvirt evacuate instance on shared storage fails

When evacuate instance which has config drive enabled on shared storage,
disk.config file can't be created because of permision error, chowns
disk.config if it already exists.

Change-Id: I20bafc3482ead30297ad5402a09e97141816f4d1
Closes-Bug: 1247526
Fix up spelling mistake

Fixes bug 1253510

This was causing a KeyError when e2label didn't work

Change-Id: I0646409cdffa4d6d5d8b2c7a9e3c300022d1c9b8
Add v3 api samples for shelve

Adds api sample tests and api samples for shelve

Partially implements blueprint v3-api-unittests
Partially implements blueprint v3-api-specification

Change-Id: Ib485ab042a340c4672ad01fb35d065fd52889b07
Add context as parameter for resume

Now for KVM when resume an instance with block storage, nova
compute will throw exception and failed to resume the VM.

The root cause is that when resume a VM with block storage,
libvirt driver needs to call conductor via rpcapi to update
block device, but the function of resume() do not have context,
this will cause RPC api failed.

Change-Id: I712777ed1d893a2b6463d30c407b0a677e37b602
Closes-Bug: #1241337
Add context as parameter for two libvirt APIs

When finish_revert_migration is called, the caller of
finish_revert_migration already includes context as a parameter,
but finish_revert_migration did not reuse this parameter and still
re-generate the context inside finish_revert_migration, we should
add context as a new parameter for finish_revert_migration so that
the functions inside of it can reuse context when needed.

For the function _create_domain_and_network in libvirt/driver.py,
it set context as an optional parameter, but context isn't really
an optional parameter for this method. We should always pass a
context down because it might be needed somewhere inside
_create_domain_and_network().

The patch includes two parts:
1) Add context as a new parameter for finish_revert_migration
2) Set context as a required parameter for _create_domain_and_network

Change-Id: I72d131fc8dce903400e3080d51ba0fc0a1320751
Closes-Bug: #1251261
Add FloatField for objects

This adds a Float field type for objects, which will be required for
things like Flavor.

Related to blueprint compute-manager-objects

Change-Id: I260a9855721046f3af24db102a24b974bfb3518a
docker: return hypervisor_version as an int rather than a string

The compute_nodes.hypervisor_version type in the database is an integer
so the docker driver needs to return the hypervisor_version as an
integer rather than a string.

Also adds the missing unit test for get_available_resource.

Closes-Bug: #1253960

Change-Id: I4e2ac13f9203b54e71f1f5cd8e15311921dd85c5
Move flags in libvirt's imagebackend to the libvirt group

This change continues moving libvirt specific flags into a libvirt
specific configuration group.

Progresses blueprint libvirt-opts-group.

DocImpact: the following flags move into the libvirt group:
    libvirt_images_type (now images_type)
    images_volume_group
    sparse_logical_volumes
    lvm_snapshot_types
    images_rbd_pool
    images_rbd_cepf_conf

Change-Id: Ic1bb8076b139787fe5acbc0574b0c75b52bf3bd7
Move flag in libvirt's utils to the libvirt group

This change continues moving libvirt specific flags into a libvirt
specific configuration group.

Progresses blueprint libvirt-opts-group.

DocImpact: the following flag is now in the libvirt group:
    snapshot_compression (formerly libvirt_snapshot_compression)

Change-Id: I9eddec5cdb72a1313e044a3c7c6d75e8ad34533a
Move flag in libvirt's vif to the libvirt group

This change continues moving libvirt specific flags into a libvirt
specific configuration group.

Progresses blueprint libvirt-opts-group.

DocImpact: the following flag has been moved into the libvirt
group:
    use_virtio_for_bridges (formerly libvirt_use_virtio_for_bridges)

Change-Id: I755139f0a470a091e36760836c9200f15aa72c37
Remove "set()" from CoreAPIMissing exception

When some core v3 API extensions are not loaded, a CoreAPIMissing
exception happens like the following example:
  CoreAPIMissing: Core API extensions are missing: set(['keypairs'])

"set()" is used only for comparing the loaded extensions and the core
extensions. It is better to remove it from the exception message.

Change-Id: Iddfa69931064a6adb6e3aa7d6efaeb28260be5e0
Fix log message format issue for api

There is a log message format typo in quota_classes api,
this patch fix it.

Change-Id: Idfb9fb16e4184e226b56bf05ffcaf9fb174feaff
Use 204 instead of 202 for delete of keypairs v3

The delete of keypairs will access db directly. It isn't async call.
So correct it to return 204.

Closes-bug: 1252169
DocImpact

Change-Id: Ie36c16e1773fa7f61b22bd6735342c7308c7ba35
Remove nullable=True in models.py which is set by default

Tiny change: nullable=True is set by default, no need to use it explicitly.

Change-Id: I28883f60a0319d4834913fcabe3a3c04410175c2
Rename instance_type parameter in migrate_disk_and_power_off to flavor

Renames the 'instance_type' parameter in the nova virt driver's
migrate_disk_and_power_off method to 'flavor' and handles the immediate
related code changes, i.e. the method itself, the implementations in the
various virt drivers, the tests, and anything called solely from
migrate_disk_and_power_off. This patch does not adjust for everything
that calls into migrate_disk_and_power_off, like from the nova compute
manager code.  That will be handled in a later patch in the series.

Part of blueprint flavor-instance-type-dedup

Change-Id: I47f2b9f6597556dca632d94f83cc1428ad73ae74
XenAPI: Fix caching of images

Clearing of the cached images' name_label means it can't be found in future.
Name label of cloned VDI should be cleared so it is not found by the
caching functions.

Main impact of this is in test scenarios where the VM is always deleted, but
a more subtle impact could be data leakage.

Change-Id: I6bb80443d31128b7d5d47e4b252db5eb8ab86940
Closes-bug: 1254714
Wrong handling of Instance expected_task_state

When Instance.save called the expected_task_state parameter can be
passed to it in order to specify what is the expected task state.
There many cases when we would expect that there no task state
for the vm (task_state=None in db). Currently the
expected_task_state=None is overwritten by the default substitution in
the save() arguments and due to this not passed correctly to the
db.instance_update_and_get_original method call.

Closes-Bug: #1225659

Change-Id: I84fd2f7e2c50c5f34c84f713d6afc040524a8078