ensure zeros are written out when clearing volumes

Note O_DIRECT is _not_ used at the end of devices to
avoid issues with odd sized blocks etc. so instead
we arrange to have dd issue an fdatasync()
to ensure the data is persisted, lest it be discarded
from the write cache when the device is unprovisioned.

* nova/virt/libvirt/utils.py (clear_logical_volume): Add 'conv=fdatasync'
to the dd option list if O_DIRECT isn't used when clearing.

Fixes bug: 1100363
Change-Id: I76789557754ebaeb6d52bb34548a2ef17808fbf6
Add api-samples to multinic extension

Partially implements bp nova-api-samples
Fixes bug 1070169

Change-Id: I923054bf92840a7fe81f78c97aa77a9201e94bdc
Parse testr output through subunit2pyunit.

When running through run_tests.sh, filter the testr output through
subunit2pyunit so that incremental progress can be seen. Should be
a step towards folks seeing incremental test progress.

Change-Id: Ic90669acabaa1c8b64aae80349bff3d3deaf59fa
allow disabling file injection completely

With the libvirt_injection_partition config set to -2,
file injection is avoided completely.

This can be required if you want to make user specified files
available from the metadata server (and config drive is not enabled),
then you would want to avoid injection completely for performance reasons,
but also to avoid boot failure if the injection itself failed.

Flags: DocImpact
Change-Id: I604f4189114515c70c58cf183622542c4633e44a
Sync latest db and importutils from oslo

Changes include:

  7c0e25a Use importutils.try_import() for MySQLdb
  33b12d3 Implement importutils.try_import.
  cf705c5 Make project pyflakes clean.

Change-Id: I448f5208919d06ca0af54d2057b8f5e042862cd6
Allow connecting to self-signed quantum endpoints

Introduce a new option 'quantum_api_insecure', matching the existing
'glance_api_insecure'. It skips the SSL validation in quantumclient.

DocImpact

Change-Id: I275517179929710bedcef5086a040b9396c0e932
Return to skipping filters when using force_hosts.

The folsom behavior was to skip filtering when using force_hosts.
This is useful for administrative tools. For example, one can verify
a disabled node by launching instances on it even though the scheduler
will ignore it for normal requests. This behavior was accidentally
changed in commit 0d9ce8319d75bfbd69fe4e0759aeed38e6054e56

Change-Id: Ib529cf8eccc4d8abe67873cfc0467f6db9d4b522
Flavor Extra Specs should require admin privileges

The previous fix added admin check in policy.json,
but code still can't recorginize the detailed actions.

This fix edited "authorize" function for flavor_extra_specs,
to make sure it will check the admin privileges in policy.json.

Also, together with the code, this fix edit old test case with admin privileges,
and added new non-admin privileges test case.

Fixes bug 1094142

Change-Id: Ia286aedb4846383ad51bd54b0984dd1feddfbf81
Default the last parameter (state) in task_log_get to None

Change the method signature to match the other definitions
of task_log_get

Fixes LP# 1117263

Change-Id: I510c05d4520c579bf07953d9ed05d7a58e051695
hypervisor-supplied-nics support in PowerVM

PowerVM does not allow MAC addresses to be directly set on LPARs.
Instead, use the hypervisor-supplied-nics feature to generate MAC
addresses we can ensure are provisioned correctly.

blueprint powervm-compute-resize-migration

Change-Id: Idad7f03d9fd61970912ec1a6978fc0af56309df4
Add regression testing tool

Tool detects if a gerrit patch contains tests to detect a regression

Usage: ./tools/regression_tester.py gerrit_review_number

Change-Id: I6353d1efda73fe5fc3098f928b6da9a28eb40590
Ban database access in nova-compute

This adds a wedge between nova-compute and the database implementation
to specifically catch, log, and deny accesses. Theoretically, this
should not be needed as a production environment would not even have
the compute nodes configured to talk to the database. However, testing
and upgraded environments may retain database access and thus avoid
hitting real issues that can be fixed up prior to release. Putting
this into the tree now will help ensure we have a consistent error
scenario for test setups prior to release.

Note that if nova is configured to use a local conductor, we do not
insert the wedge, which provides an easy out for anyone needing to
get a production system past a missed database access.

Related to bp no-db-compute.

Based on Russell's original tracer hack:

   I328fa92d5bfdadd5022f5c7efe981396d8ae7962

Change-Id: I478230220633e0d2ff94b6a4d756e07eab8517d7
Fix lazy load 'system_metadata' failed problem.

When using lazy load of SQLAlchemy, parent instance may have
a problem not bouding to a session object and fail the lazy loading
system_metadata.
Using joinedload to eagly load the system_metadata in
_instance_get_all_query to fix the problem.

Fixes: bug 1117232
Change-Id: Id24ca534f6eb57167aee22aca12c1274c90baf3c
Update modules from common required for rpc with lock detection.

This change updates the versions of the local, lockutils and rpc
modules from common. This is required so that I can implement
automated detection of rpcs being made while holding locks, which
will come in a later review.

Sneaks up on bug 1063222.

Change-Id: I3689c133d1efc7425a44662abbdb434b5f36c64e
Recache or rebuild missing images on hard_reboot

The primary purpose of this change is to provide the ability to
re-cache missing backing files on hard_reboot. The old
pre_block_migration function was already performing a very similar
operation. That function has been refactored to be idempotent and
renamed to _create_images_and_backing. The pre_block_migration
function is a wrapper, with some additional checking, around the
renamed function.

Image backend was also adjusted to look for either a missing backing
file or disk image, recaching or creating accordingly. It should
also be idempotent, never clobbering existing images.

Change-Id: Icf4c488d6db59e732b463d08d0606b428ee1e7b9
Support hypervisor supplied macs in nova-network

When hypervisors supply a set of MAC addresses for instances, have
nova-network use the supplied values when creating VIFs.

This function is required for the PowerVM driver to be properly
supported.

Change-Id: I922797af6b2b829561f61095bef2098c2d31cde5
Extension for rebuild-for-ha

Adds an extension for blueprint rebuild-for-ha

Evacuation of VM from a failed host for HA.
New api supports recreate/rebuild modes. With shared storage,
VM is recreated on the target from disk.
Otherwise, rebuild from original image is performed.

DocImpact

Change-Id: Id7e3e68dbaaf74e9314440d98195a92454078fd9
Co-authored-by: Oshrit Feder <oshritf@il.ibm.com>
Merge LibvirtHybridOVSBridgeDriver into LibvirtGenericVIFDriver

The LibvirtHybridOVSBridgeDriver VIF driver and the
LibvirtOpenVswitchDriver VIF driver both support OpenVswitch
based guest configs. The difference is that the hybrid driver
does not connect the VM directly to the OVS bridge. Instead it
connects the guest to a private Linux bridge, connected to the
OVS bridge via a veth pair. This allows iptables firewall rules
to be applied to the guest VIF. The hybrid driver should always
be used unless the admin has disabled Nova firewalls using the
NoopFirewallDriver impl. Rather than require the admin to figure
this out themselves, pick the right impl automatically based
on the configured firewall driver.

When Quantum gains support for firewalling, it will need to be
able to inform Nova whether it has got a filtering applied to
the VIFs, and thus let Nova intelligently skip its own attempt
at filtering.

The LibvirtGenericVIFDriver class can use the 'vif_type' mapping
field to determine whether an OVS network configuration is
required. In combination with a check against the firewall
driver, a dedicated driver for OVS hybrid setup is no longer
required.

The LibvirtHybridOVSBridgeDriver class functionality is
merged into LibvirtGenericVIFDriver. For backwards compatibility
with the Folsom release, the existing LibvirtHybridOVSBridgeDriver
class is made to inherit from LibvirtGenericVIFDriver and directly
call the bridge specific setup APIs. This eases migration to
the new VIF impl during the Grizzly deployment lifecycle, with
an expectation that the LibvirtHybridOVSBridgeDriver stub
will be deleted after the Havana release.

Bug: 1050433
Blueprint: libvirt-vif-driver
Change-Id: I93294409a75aaa15d297471ab3e600c3c124bcde
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Introduce support for 802.1qbg and 802.1qbh to Nova VIF model

Define the nova network VIF model properties for supporting
802.1qbg / 802.1qbh networking in hypervisor drivers. Also
wire up support to the libvirt driver for these new modes

Blueprint: libvirt-vif-driver
Change-Id: I91039790900170bfd698ea484d5f8a22445806ae
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Access DB as dict not as attributes part 5

Partially implements bp db-api-cleanup
Change-Id: Id34c20701ae0cae503c2218c9a6008d5e0bb0e76