ensure zeros are written out when clearing volumes

Note O_DIRECT is _not_ used at the end of devices to
avoid issues with odd sized blocks etc. so instead
we arrange to have dd issue an fdatasync()
to ensure the data is persisted, lest it be discarded
from the write cache when the device is unprovisioned.

* nova/virt/libvirt/utils.py (clear_logical_volume): Add 'conv=fdatasync'
to the dd option list if O_DIRECT isn't used when clearing.

Fixes bug: 1100363
Change-Id: I76789557754ebaeb6d52bb34548a2ef17808fbf6
Add api-samples to multinic extension

Partially implements bp nova-api-samples
Fixes bug 1070169

Change-Id: I923054bf92840a7fe81f78c97aa77a9201e94bdc
Parse testr output through subunit2pyunit.

When running through run_tests.sh, filter the testr output through
subunit2pyunit so that incremental progress can be seen. Should be
a step towards folks seeing incremental test progress.

Change-Id: Ic90669acabaa1c8b64aae80349bff3d3deaf59fa
allow disabling file injection completely

With the libvirt_injection_partition config set to -2,
file injection is avoided completely.

This can be required if you want to make user specified files
available from the metadata server (and config drive is not enabled),
then you would want to avoid injection completely for performance reasons,
but also to avoid boot failure if the injection itself failed.

Flags: DocImpact
Change-Id: I604f4189114515c70c58cf183622542c4633e44a
Sync latest db and importutils from oslo

Changes include:

  7c0e25a Use importutils.try_import() for MySQLdb
  33b12d3 Implement importutils.try_import.
  cf705c5 Make project pyflakes clean.

Change-Id: I448f5208919d06ca0af54d2057b8f5e042862cd6
Allow connecting to self-signed quantum endpoints

Introduce a new option 'quantum_api_insecure', matching the existing
'glance_api_insecure'. It skips the SSL validation in quantumclient.

DocImpact

Change-Id: I275517179929710bedcef5086a040b9396c0e932
Return to skipping filters when using force_hosts.

The folsom behavior was to skip filtering when using force_hosts.
This is useful for administrative tools. For example, one can verify
a disabled node by launching instances on it even though the scheduler
will ignore it for normal requests. This behavior was accidentally
changed in commit 0d9ce8319d75bfbd69fe4e0759aeed38e6054e56

Change-Id: Ib529cf8eccc4d8abe67873cfc0467f6db9d4b522
Flavor Extra Specs should require admin privileges

The previous fix added admin check in policy.json,
but code still can't recorginize the detailed actions.

This fix edited "authorize" function for flavor_extra_specs,
to make sure it will check the admin privileges in policy.json.

Also, together with the code, this fix edit old test case with admin privileges,
and added new non-admin privileges test case.

Fixes bug 1094142

Change-Id: Ia286aedb4846383ad51bd54b0984dd1feddfbf81
Default the last parameter (state) in task_log_get to None

Change the method signature to match the other definitions
of task_log_get

Fixes LP# 1117263

Change-Id: I510c05d4520c579bf07953d9ed05d7a58e051695
hypervisor-supplied-nics support in PowerVM

PowerVM does not allow MAC addresses to be directly set on LPARs.
Instead, use the hypervisor-supplied-nics feature to generate MAC
addresses we can ensure are provisioned correctly.

blueprint powervm-compute-resize-migration

Change-Id: Idad7f03d9fd61970912ec1a6978fc0af56309df4
Add regression testing tool

Tool detects if a gerrit patch contains tests to detect a regression

Usage: ./tools/regression_tester.py gerrit_review_number

Change-Id: I6353d1efda73fe5fc3098f928b6da9a28eb40590
Fix inject_files for storing binary file.

Now a nova-api server decodes injected_files by base64 after receiving
encoded files from nova-client.
And the server fails to publish run_instance because utf8 codec can't
decode due to binary data.

This patch fixes this problem, because the base64 decode is operated when
a nova-compute server receives injected_files from a nova-api server.

Fixes bug 1112134

Change-Id: I664134c74893a970de4464161071692a5b2b5ce7
Ban database access in nova-compute

This adds a wedge between nova-compute and the database implementation
to specifically catch, log, and deny accesses. Theoretically, this
should not be needed as a production environment would not even have
the compute nodes configured to talk to the database. However, testing
and upgraded environments may retain database access and thus avoid
hitting real issues that can be fixed up prior to release. Putting
this into the tree now will help ensure we have a consistent error
scenario for test setups prior to release.

Note that if nova is configured to use a local conductor, we do not
insert the wedge, which provides an easy out for anyone needing to
get a production system past a missed database access.

Related to bp no-db-compute.

Based on Russell's original tracer hack:

   I328fa92d5bfdadd5022f5c7efe981396d8ae7962

Change-Id: I478230220633e0d2ff94b6a4d756e07eab8517d7
Fix lazy load 'system_metadata' failed problem.

When using lazy load of SQLAlchemy, parent instance may have
a problem not bouding to a session object and fail the lazy loading
system_metadata.
Using joinedload to eagly load the system_metadata in
_instance_get_all_query to fix the problem.

Fixes: bug 1117232
Change-Id: Id24ca534f6eb57167aee22aca12c1274c90baf3c
Update modules from common required for rpc with lock detection.

This change updates the versions of the local, lockutils and rpc
modules from common. This is required so that I can implement
automated detection of rpcs being made while holding locks, which
will come in a later review.

Sneaks up on bug 1063222.

Change-Id: I3689c133d1efc7425a44662abbdb434b5f36c64e
Cells: Add cells support to instance_usage_audit_log api extension

Adds task_log_get_all() call to HostAPI() so that we can use the local
DB when cells is disabled or proxy via cells if cells is enabled.

Adds task_log_get_all() call to cells.

Change-Id: I9e78c17fcf70f98903d0a26c1de3e2581b8977ad
Cells: Add cells support to hypervisors extension

This removes the DB calls from the extension and moves them into
HostAPI().  The cells version of HostAPI() proxies them to the
appropriate cell or cells.

Change-Id: I17ff10e62d2f10b85a557edcc1509ca3b0d15a29
Recache or rebuild missing images on hard_reboot

The primary purpose of this change is to provide the ability to
re-cache missing backing files on hard_reboot. The old
pre_block_migration function was already performing a very similar
operation. That function has been refactored to be idempotent and
renamed to _create_images_and_backing. The pre_block_migration
function is a wrapper, with some additional checking, around the
renamed function.

Image backend was also adjusted to look for either a missing backing
file or disk image, recaching or creating accordingly. It should
also be idempotent, never clobbering existing images.

Change-Id: Icf4c488d6db59e732b463d08d0606b428ee1e7b9
Support hypervisor supplied macs in nova-network

When hypervisors supply a set of MAC addresses for instances, have
nova-network use the supplied values when creating VIFs.

This function is required for the PowerVM driver to be properly
supported.

Change-Id: I922797af6b2b829561f61095bef2098c2d31cde5
Extension for rebuild-for-ha

Adds an extension for blueprint rebuild-for-ha

Evacuation of VM from a failed host for HA.
New api supports recreate/rebuild modes. With shared storage,
VM is recreated on the target from disk.
Otherwise, rebuild from original image is performed.

DocImpact

Change-Id: Id7e3e68dbaaf74e9314440d98195a92454078fd9
Co-authored-by: Oshrit Feder <oshritf@il.ibm.com>