Enable compute_node_update to tolerate deadlocks

When running the CI there were cases when the aformentioned method would
throw a DBDeadlock exception.

Change-Id: I98d1a804c51e1bf3bb96193d82ffe7e5d064e134
Closes-bug: #1253455
Ignore duplicate delete requests

Stop delete requests from queueing up in the compute manager if the
task_state is already set to DELETING. This prevents exhausting thread
pools when the service is stuck on a lock/external request.

Dealing with delete requests that never got executed is not in scope of
this change and will be submitted separately.

Change-Id: I2f97f93bd714e0ea3b6d4fa3ac457ab43eed00e1
Partial-Bug: #1248563
Cleanup 'deleting' instances on restart

In case some instance was marked as deleting but for some reason didn't
finish (for example request was stuck in libvirt), retry the delete at
startup. This happens at startup and on the host owning the instance,
so there's no reason to use the lock.

Change-Id: Iad18e9a7c6cb8e272e67a82284127ad895441dcf
Partial-Bug: #1248563
Closes-Bug: 1247174
Implements: blueprint recover-stuck-state
Add API schema for v3 multinic API

By defining the API schema, it is possible to separate the validation
code from the API method. The API method can be more simple.
In addition, a response of API validation error can be consistency for
the whole Nova API.

Partially implements blueprint nova-api-validation-fw

Change-Id: I62a55c498a1164ff56f82495f720cba67da8483b
Add preserve_ephemeral option to rebuild

The preserve_ephemeral option for rebuild will preserve the content of
the ephemeral partition, making stateful golden image based
deployments possible even when clouds haven't deployed Cinder, or the
hypervisor doesn't support Cinder (e.g. BareMetal / Ironic).

Partial-Bug: #1174154
blueprint: baremetal-preserve-ephemeral
Co-Authored-By: Robert Collins <rbtcollins@hp.com>

Change-Id: Id33d5d4107f89814842a3f0b7f33690dd7e3aadc
Show bm deploy how to preserve ephemeral content.

This changes the baremetal deploy helper to be able to preserve the
ephemeral partition, if the deploy requests that. The next patch in
the series makes it possible to store that intent in the deploy and
connects it to the deploy() method.

Partial-Bug: #1174154
blueprint: baremetal-preserve-ephemeral
Change-Id: I6bf01e52589c5894eb043f2b57e915d52e81ebc3
Support preserve_ephemeral in baremetal.

We need to be able to capture the semantic intent of a rebuild with
ephemeral preservation for the baremetal helper, which means another
column in bm.modes.

Partial-Bug: #1174154
blueprint: baremetal-preserve-ephemeral
Co-Authored-By: Roman Podoliaka <rpodolyaka@mirantis.com>
Change-Id: Icf627f38dac02b7d06ff0f1816741e5513a4fcf3
Add API schema for v3 evacuate API

By defining the API schema, it is possible to separate the validation
code from the API method. The API method can be more simple.
In addition, a response of API validation error can be consistent for
the whole Nova API.

Partially implements blueprint nova-api-validation-fw

Change-Id: I7587fd8c27b820b2a9b751f9c9ae30da0aa41ec7
remove redundant __init__() overwriting when getting ExtensionResources

As described in #1266344, some of child class in
<nova>/nova/api/openstack/compute/contrib/ has overwrited __init__()
without specified needs. Removing that could help keeps codes clean and
consistent (nearly all others don't overwrite __init__()).

Change-Id: I87dfd9c1622bc8250f620ef591391a0e98573f78
Closes-Bug: #1266344
disable debug in eventlet.wsgi server

the debug flag basically just means that we get lines like:

 ....wsgi.server [-] (6966) accepted ('127.0.0.1', 51944)

After *every single* wsgi request, filling up the logs with
completely irrelevant data at info level because eventlet.wsgi
does not use python logging.

Change-Id: I508b963a0c84e17df845aad260dd8ee157abbea8
Add host and details column to instance_actions_events table

Instance action event could be a useful tool to understand and trouble
shooting the scene after invoking an API request (action) for an
instance.

This change extends the action event model by adding additional two
columns to record on which host this event took place and more detail
message for a specific event.

Implements: blueprint improve-instance-action-events
Change-Id: Id209cd7b8a3a41f86cb802d4dd681443d5717bd0
Add explicit discussion of dependencies to README.rst

Noted that building the documentation has more dependencies
than merely running Nova.
Noted that the standard development environment is sufficient.
Noted that the standard runtime environment plus oslo.sphinx
is sufficient.

Change-Id: I37b84a1bdfbc906b005c0a53c2c570c0695937b3
Closes-Bug: #1266623
Hide VIR_CONNECT_BASELINE_CPU_EXPAND_FEATURES where needed

Running unit test against a newer libvirt fails due to
VIR_CONNECT_BASELINE_CPU_EXPAND_FEATURES being always available.

This was mostly hidden as the old unit test globally deleted
the attribute during execution. Removing this side effect
and use mocking of attributes instead where needed.

Related-Bug: LP #1266534

Change-Id: Id0e6d8771f1a43f402b6323f0c0df38ee23e0fa8
Fix docstring in libvirt.driver.LibvirtDriver.get_instance_disk_info()

Update param list to match reality and fold in pre-existing
NOTEs about the parameters into the doc string.

Change-Id: I994f75ef4d0f1540c007ec1d430c6973e9308e35
Move fake_network config option to linux_net

Option 'fake_network' is used by nova/network/linux_net.py but not
imported from nova/network/manager.py. Both use it, but manager imports
linux_net (via nova/network/api.py + nova/network/floating_ips.py) so it
makes more sense to define the option in linux_net and import_opt in
manager.

Precise example import chain:
    bin/nova-manage
    nova/cmd/manage.py: import_opt('flat_network_bridge', 'nova.network.manager')
    nova/network/manager.py: from nova.network import api as network_api
      -- At this point, manager has started to import but not yet called
      register_opts
    nova/network/api.py: from nova.network import floating_ips
    nova/nova/network/floating_ips.py: import_opt('public_interface', 'nova.network.linux_net')
    nova/nova/network/linux_net.py: import_opt('fake_network', 'nova.network.manager')
      -- BANG! Manager hasn't registered the option we want to import

Change-Id: I66f021b34aef650dca148a58203c249f6aa0e83e
Small edits on help strings

These were initially noted during import into the manuals at
https://review.openstack.org/67502

Fixes include rework of text, missing or extra spaces, wrong
capitalization and no sentence style capitalization.

A hacking check has been added that tests that all config options
start with a capital letter.

Change-Id: I0a1c333bbb3a812932d3972a7403f94da0bff341
Co-Authored-By: Daniel Berrange <berrange@redhat.com>
Co-Authored-By: Diane Fleming <diane.fleming@rackspace.com>
XenAPI: get_info was very expensive

Getting the whole VM record is expensive and slowed down Tempest tests
significantly.  Replaced with getting each required item.

When getting power state, only need that specific value, so reduce the
calls to get_info too.

Change-Id: I93bcef5b692a531aac41518dc29d5d4f46cded65
Additional check for qemu-nbd hang

/sys/block/*device*/pid check is not enough. I see that the unix
socket used by the device my be stuck as well, so let's add another
check for the path to the unix socket for the device as well to
figure out if the device is free. Complain loud and clear that the
qemu-nbd is leaking resources.

Change-Id: I28cedffba7a9915ef6f7888989e40e4d0de475c6
Closes-Bug: #973413
Partial-Bug: #1254890
libvirt: Fix root disk leak in live mig

This patch makes sure that i_create_images_and_backing method of the
libvirt driver (called in several places, but most problematic one is
the call in the pre_live_migration method) creates all the files the
instance needs that are not present.

Prioir to this patch - the method would only attempt to download the
image, and if it did so with the path of the ephemeral drives, it could
expose the image to other users as an ephemeral devices. See the related
bug for more detaiis.

After this patch - we properly distinguish between image, ephemeral and
swap files, and make sure that the imagebackend does the correct thing.

Closes-bug: #1251590

Co-authored-by: Loganathan Parthipan <parthipan@hp.com>

Change-Id: I78aa2f4243899db4f4941e77014a7e18e27fc63e
Allow delete when InstanceInfoCache entry is missing

The existing code wouldn't allow you to delete 'broken' instances that were
(for whatever reason) missing their InstanceInfoCache entry. This patch fixes
it such that those instances can be deleted.

Adds a test case to cover the new exception handling.

Closes-Bug: #1266919

Co-Authored-By: Matt Riedemann <mriedem@us.ibm.com>
Change-Id: I3d566a25e340b5d5de86740141e1eaa5b3742915