Make hypervisor_version an int in fakeVirt driver

This is a follow up to 9e770e62135fe9c2c8ac0121a5a79245b25a7847, which
missed the change in a duplicate copy of host_status.

Add regression test to test_virt_drivers.

This requires changes to the API samples, because they were wrong. virt
drivers use convert_version_to_int which converts a version string to a
4 digit number ("1.0" becomes to 1000)

Change-Id: I28ce23509e3c9feae183a49a8fc5bf3c7c601295
Closes-Bug: #1285035
VM in rescue state must have a restricted set of actions

Right now it is possible to pause, suspend and stop a VM in state RESCUED,
so after the state is changed, it's not possible to trigger unrescue anymore
since the original state is lost.

This patch remove vm_states.RESCUED as valid state from stop,
pause and suspend actions.

The vm_states devref is also updated to reflect this change including the
current reboot flow.( vm_states.RESCUED cannot be rebooted as per
today code)

DocImpact
Closes-Bug: #1319182
Co-Authored-By: Cyril Roelandt <cyril.roelandt@enovance.com>
Change-Id: I531dea5a5499bf93c24bea37850d562134dee281
VMware: remove unused parameter 'network_info'

The parameter 'network_info' is not used for destroying an instance and
doesn't need to be passed through the whole stack.

Change-Id: I0a8b912ff7693478917e849e17093d1541adeeeb
Libvirt: Added suffix to configdrive_path required for rescue

It was observed that during nova rescue, nova failed to create
a rescue disk for disk.config. The reason being that .rescue
suffix was missing from the configdrive path. It was working
till Havana but the functionality broke in Icehouse. This commit
fixes the suffix problem.

Unittests have been added to verify following scenarios:
    1. Make sure that .rescue disks are created when not using
        config drive
    2. Make sure that .rescue disks are created when using
        config drive

Closes-Bug: #1334024

Change-Id: I87449ffddd047cb84b7b881757ea4c29927b95da
Change the comments of SOFT_DELETED race condition

_reclaim_queued_deletes will only find instances in SOFT_DELETED
state and delete them if they are only old enough. The quotas will be
committed when the instance was soft-deleted so that resource can
be used right after the soft-delete operation.

There are some concern about the quota inconsistency for the operation,
The only case that the quota might be inconsistent is
the compute node died between set instance state to SOFT_DELETED
and quota commit to DB; when compute node start again
it will have no idea the reservation is committed or not or even
expired, since it's a rare case, so marked as todo.

There are some alternatives if the problem need to be fixed, e.g.
able to find reservation when nova compute restart, so we can double
check whether the quota is committed or not.

Partial-Bug: #1296414

Change-Id: Idf9c179b2dd439462a646568ffd5098cd5d7851f
Fix virt BDM __setattr__ and __getattr__

Previously we would only allow these to work on fields that are found in
the _proxy_as_attr collection of the object and we would unconditionally
raise an AttributeError otherwise.

This behaviour is too restrictive as it prevents us from using mock on
these objects properly, Instead we now delegate back to the parent
class for a more natural behaviour.

Change-Id: I5c221eef8f05c66ac2e2fb4abf645748407462c0
Format eph disk with  specified format in libvirt

novaclient has following command parameters:
--ephemeral size=<size>[,format=<format>]
Create and attach a local ephemeral block device of
<size> GB and format it to <format>.

so mkfs should use the specified format instead of using
default format.

Change-Id: I7a8753284d7b1da1a1203e85b430bd0c5012937a
Closes-Bug: #1280132
add get_by_metadata_key to AggregateList object

This change adds the get_by_metadata_key function to the AggregateList object
and the aggregate_get_by_metadata_key function to the DB API.

Previously, the only DB API function available for finding aggregates matching
a metadata key returned a dictionary of aggregated metadata. The
aggregate_get_by_metadata_key function returns all rows matching the specified
metadata key, and is called by the existing aggregate_host_get_by_metadata_key
DB API function.

The get_by_metadata_key function returns a list of Aggregate objects matching
the specified metadata key.

Related to blueprint compute-manager-objects-juno

Change-Id: If50c9f613dd192ab44577f26a81dd5b40e3a7af7
libvirt+lxc: Unmount guest FS from host on error

If an error occurs during `_create_domain`, we need to ensure that the guest's
FS is unmounted from the host in all cases.

This is necessary because, if we leave the FS mounted to the host, the
`lvremove` triggered by `delete` won't work because the filesystem will
'still be in use'.

The solution is to wrap the code in a `try/finally` to ensure the cleanup
routines (which unmount) are always called.

Change-Id: If863cf813dddc1e4554fb87b945c68b75b25f9a2
Closes-Bug: 1333827
libvirt: Support IPv6 with LXC

Libvirt's LXC implementation exposes a read-only  `/proc/sys/net` to the
guests. This causes some of the guest's default network configuration scripts
to fail.

This patch works-around the issue by using `post-up` hooks to configure IPv6.

Closes-Bug: 1340791

Change-Id: I805cad98d855fcb2c90b07e98ad3653d1620bd42
xenapi: Use netuils.get_injected_network_template

The Xen driver duplicates quite a bit of code used to inject the `interfaces`
template into the guest.

This patch makes the Xen driver use the generic `netutils` version.

Parameter defaults were moved from the signature into the method since in the
original code, you wouldn't be able to override them due to their value being
populated at import-time, not run-time.

Change-Id: I4cd1631b8e6ec2dc3fae1f8eb0362aecead53f1e
Update requirements to include decorator>=3.4.0

This third party package is used but there was no specific
requirement for it.

Change-Id: I50fcc7a1c4029b36e02e75eaf27df7ca791aef4e
Xen: Cleanup orphan volume connections on boot failure

If the boot process fails after the VDI creation but before a VBD is
created then the current cleanup methods will not work as they all rely
on lookups via VBD.

Change-Id: Id0d93ee60f75bf319baf7859b220ca325175128a
Closes-bug: #1329941
Enable live migration unit test use instance object

test_post_live_migration_no_shared_storage_working_correctly is
still using instance dict, we should change it to using instance
object.

Related to blueprint compute-manager-objects-juno

Change-Id: I7709b32ebeef88eb9759d0275c3076091b6be669
VMware: support the hotplug of a neutron port

Nova already has support for hotplugging neutorn ports in the libvirt
driver. This extends the support to the VMware driver.

Implements the blueprint vmware-hot-plug

Change-Id: I01b78621016e20c8e61d48bdeeac166f08f799be
VMware: Remove unnecessary deepcopy()s in test_configdrive

test_configdrive was doing a deepcopy of its test_instance object in
some tests. This was unnecessary, as the object is created fresh in
setUp().

Also, change a couple of dict lookups of test_instance to object
properties.

Change-Id: I3a5039fde9f8fba5285d90b28743b7af8cd1a168
libvirt: remove unused list_instance_ids method

All code has been converted to use list_instance_domains so the
list_instance_ids method can finally be removed.

Blueprint: libvirt-domain-listing-speedup
Change-Id: I9ce33cda5608796e0eec4ec0804ee179a4704d9d
libvirt: remove flawed get_num_instances method impl

The get_num_instances method impl simply asks libvirt
how many running domains there are. This is flawed for
a number of reasons.

 - It doesn't filter out 'Domain-0' on Xen
 - It doesn't count inactive instances

As a result the value is returns is inconsistent with
the number of instances shown by 'list_instances'.
This will cause the _sync_power_states method in the
compute manager to constantly try to re-sync the list
of instances.

Remove the flawed impl and just allow the base class
default impl to be used. This just does a len() call
over the return of 'list_instances'. Previously this
was quite a heavy API call, but with new enough libvirt
it can be completed in a single API call which should
be fine.

Blueprint: libvirt-domain-listing-speedup
Change-Id: I96242717afa5c928f1cfb5f307b71739f2147e69
Fix Cells ImagePropertiesFilter can raise exceptions

In a misconfigured scenario the ImagePropertiesFilter can raise an
exception which causes the image to be unbuildable in the region. This
happens when the capabilities is set for prominent_hypervisor_version
to an empty list.

Change-Id: I17bd9d87fa69ad34b19591018543a34dfd050978
Closes-Bug: 1343462
Do not process events for instances without host

In some cases Neutron might send events such as 'VIF unplugged'
for instances which are either being deleted or shelved. When
that happens there will be a failure in dispatching the event
to the appropriate compute node - as there is no host for the
instance.

As multiple neutron events can be stashed in a single call
it is important to avoid that this kind of errors will prevent
processing of other events in the same call.

This patch does not process events for instances without a host,
marking them as failed.

When the above condition occurs, the create event request will
return a 207 response code. For specific events, a 422
unprocessable entity code will be set.

This patch also preserve the characteristic that events are
returned in the response in the same order they were found in
the request.

Change-Id: I18062b81e50c722ec96b4296ac39384493683ae3
Closes-Bug: #1333654