Change the comments of SOFT_DELETED race condition

_reclaim_queued_deletes will only find instances in SOFT_DELETED
state and delete them if they are only old enough. The quotas will be
committed when the instance was soft-deleted so that resource can
be used right after the soft-delete operation.

There are some concern about the quota inconsistency for the operation,
The only case that the quota might be inconsistent is
the compute node died between set instance state to SOFT_DELETED
and quota commit to DB; when compute node start again
it will have no idea the reservation is committed or not or even
expired, since it's a rare case, so marked as todo.

There are some alternatives if the problem need to be fixed, e.g.
able to find reservation when nova compute restart, so we can double
check whether the quota is committed or not.

Partial-Bug: #1296414

Change-Id: Idf9c179b2dd439462a646568ffd5098cd5d7851f
Format eph disk with  specified format in libvirt

novaclient has following command parameters:
--ephemeral size=<size>[,format=<format>]
Create and attach a local ephemeral block device of
<size> GB and format it to <format>.

so mkfs should use the specified format instead of using
default format.

Change-Id: I7a8753284d7b1da1a1203e85b430bd0c5012937a
Closes-Bug: #1280132
add get_by_metadata_key to AggregateList object

This change adds the get_by_metadata_key function to the AggregateList object
and the aggregate_get_by_metadata_key function to the DB API.

Previously, the only DB API function available for finding aggregates matching
a metadata key returned a dictionary of aggregated metadata. The
aggregate_get_by_metadata_key function returns all rows matching the specified
metadata key, and is called by the existing aggregate_host_get_by_metadata_key
DB API function.

The get_by_metadata_key function returns a list of Aggregate objects matching
the specified metadata key.

Related to blueprint compute-manager-objects-juno

Change-Id: If50c9f613dd192ab44577f26a81dd5b40e3a7af7
libvirt+lxc: Unmount guest FS from host on error

If an error occurs during `_create_domain`, we need to ensure that the guest's
FS is unmounted from the host in all cases.

This is necessary because, if we leave the FS mounted to the host, the
`lvremove` triggered by `delete` won't work because the filesystem will
'still be in use'.

The solution is to wrap the code in a `try/finally` to ensure the cleanup
routines (which unmount) are always called.

Change-Id: If863cf813dddc1e4554fb87b945c68b75b25f9a2
Closes-Bug: 1333827
xenapi: Use netuils.get_injected_network_template

The Xen driver duplicates quite a bit of code used to inject the `interfaces`
template into the guest.

This patch makes the Xen driver use the generic `netutils` version.

Parameter defaults were moved from the signature into the method since in the
original code, you wouldn't be able to override them due to their value being
populated at import-time, not run-time.

Change-Id: I4cd1631b8e6ec2dc3fae1f8eb0362aecead53f1e
libvirt: remove flawed get_num_instances method impl

The get_num_instances method impl simply asks libvirt
how many running domains there are. This is flawed for
a number of reasons.

 - It doesn't filter out 'Domain-0' on Xen
 - It doesn't count inactive instances

As a result the value is returns is inconsistent with
the number of instances shown by 'list_instances'.
This will cause the _sync_power_states method in the
compute manager to constantly try to re-sync the list
of instances.

Remove the flawed impl and just allow the base class
default impl to be used. This just does a len() call
over the return of 'list_instances'. Previously this
was quite a heavy API call, but with new enough libvirt
it can be completed in a single API call which should
be fine.

Blueprint: libvirt-domain-listing-speedup
Change-Id: I96242717afa5c928f1cfb5f307b71739f2147e69
Do not process events for instances without host

In some cases Neutron might send events such as 'VIF unplugged'
for instances which are either being deleted or shelved. When
that happens there will be a failure in dispatching the event
to the appropriate compute node - as there is no host for the
instance.

As multiple neutron events can be stashed in a single call
it is important to avoid that this kind of errors will prevent
processing of other events in the same call.

This patch does not process events for instances without a host,
marking them as failed.

When the above condition occurs, the create event request will
return a 207 response code. For specific events, a 422
unprocessable entity code will be set.

This patch also preserve the characteristic that events are
returned in the response in the same order they were found in
the request.

Change-Id: I18062b81e50c722ec96b4296ac39384493683ae3
Closes-Bug: #1333654
Correctly reject request to add lists of hosts to an aggregate

Attempting to add or remove a list of hosts to an aggregate
results in an internal server error. This patch fixes the input
validation so such a request is correctly rejected and a 400
error returned.

Change-Id: I760db36d685cff20b01a72160b8b3fbb8a70b412
Closes-Bug: #1321653
Correct use of nova.image.glance in compute API

The nova.api.openstack.compute.images modules was using the
nova.image.glance module directly, instead of using the new,
standardized nova.image.API module.

This patch corrects that usage and updates the associated unit tests to
use mock instead of the nova.tests.api.fake.stub_out_glance() method.

Copies the nova.tests.api.fake._make_image_fixtures() call to a new
nova.tests.image_fixtures module, and corrects a number of errors in the
original function, including not returning datetime objects for
timestamp fields (as the nova.image.API returns) and ensuring that the
ID field is a string in the returned fixture dicts.

The _make_image_fixtures() function and the stub_out_glance() function
will be removed in the next patch, which fixes up the image_metadata
module, which is the last unit test that uses them.

Change-Id: Icd22ff3f438b8ba476f2637c7d4129ea6d0f45e8
Add extensible resources to resource tracker

A resource plugin extension point is added to the resource
tracker to allow the types of resources allocated at the compute
node to be extensible. Information maintained by these plug-ins
is written to the compute_nodes table in the database. The
scheduler uses the information in the compute_nodes table to
determine scheduling decisions.

A plugin that implements vcpu resource tracking is included
and all other code for tracking vcpu has been removed. This
example ensures the plugins are tested in gate jobs.

Co-Authored-By: Andrea Rosa  <andrea.rosa@hp.com>
Co-Authored-By: Paul Murray  <pmurray@hp.com>
This is part of: blueprint extensible-resource-tracking

Change-Id: I64108338e3c958ba1276aaf113a68861cbe286f5
Re-add H803 to flake8 ignore list.

I6c2537dd27c947e36ebf37eb3b5c8a1ab8b026a1 mistakenly removed H803 since
running flake8 had no hits for this test, but H803 is the period in the
commit message which we are explicitly ignoring until the next version
of hacking is released (which will remove this check).

Change-Id: I22860cc61336ef5bf8ce4509fe130160bdf7e45b
Fix unit tests related to cloudpipe_update

In some unit tests related to cloudpipe_update, there are two negative
factors(ex. bad url and bad body) and current tests don't verify its
purposes. This patch removes unnecessary factor.

Change-Id: I5e78b47e6f4c7359a7f0c16898a26e3649d92eb2
Fix error status code for cloudpipe_update

When passing bad body in a request, most APIs return BadRequest response.
However, cloudpipe_update API doesn't do it.
This patch fixes the error status code and adds a unit test related to
this change.

Change-Id: I16a48e6913f52baa870e04196529fda619c03098
Fix typo

Change-Id: I674512d817d44897391e62439e1730bdf89ca9b1
Remove unneeded calls in test_shelve to start instances

During running of tox the system can timeout on slower systems.  This
is due to rpc calls to other threads failing in testing.  This
particular case involved shelving calls to start instances.  However,
when we started using objects, these calls became unnecessary for these
unit tests.  Thus we are removing the calls which should remove
the traceback for slow systems.

Change-Id: I52f1e86815cc2cac8dc31865f00eef2d4ffeea59
Partial-Bug: 1311778
Updated from global requirements

Change-Id: Ie52204a9bbfd3e608a947f71dd3f406129aa9533
Merge "Do not process events for instances without host"
Merge "xenapi: Use netuils.get_injected_network_template"
Return status for compute node

Currently when return compute node information, there is no status returned.

When the corresponding service is disabled or down and users try to
do 'hypervisor-list' or 'hypervisor-show', they will have no idea of it.

Implements: blueprint return-status-for-hypervisor-node
Closes-Bug: #1285259

DocImpact

Change-Id: I17c53b454ccef023f298f1b8875daef965d2325d
Merge "Fix typo"