ensure zeros are written out when clearing volumes

Note O_DIRECT is _not_ used at the end of devices to
avoid issues with odd sized blocks etc. so instead
we arrange to have dd issue an fdatasync()
to ensure the data is persisted, lest it be discarded
from the write cache when the device is unprovisioned.

* nova/virt/libvirt/utils.py (clear_logical_volume): Add 'conv=fdatasync'
to the dd option list if O_DIRECT isn't used when clearing.

Fixes bug: 1100363
Change-Id: I76789557754ebaeb6d52bb34548a2ef17808fbf6
Add api-samples to multinic extension

Partially implements bp nova-api-samples
Fixes bug 1070169

Change-Id: I923054bf92840a7fe81f78c97aa77a9201e94bdc
Parse testr output through subunit2pyunit.

When running through run_tests.sh, filter the testr output through
subunit2pyunit so that incremental progress can be seen. Should be
a step towards folks seeing incremental test progress.

Change-Id: Ic90669acabaa1c8b64aae80349bff3d3deaf59fa
allow disabling file injection completely

With the libvirt_injection_partition config set to -2,
file injection is avoided completely.

This can be required if you want to make user specified files
available from the metadata server (and config drive is not enabled),
then you would want to avoid injection completely for performance reasons,
but also to avoid boot failure if the injection itself failed.

Flags: DocImpact
Change-Id: I604f4189114515c70c58cf183622542c4633e44a
Sync latest db and importutils from oslo

Changes include:

  7c0e25a Use importutils.try_import() for MySQLdb
  33b12d3 Implement importutils.try_import.
  cf705c5 Make project pyflakes clean.

Change-Id: I448f5208919d06ca0af54d2057b8f5e042862cd6
Allow connecting to self-signed quantum endpoints

Introduce a new option 'quantum_api_insecure', matching the existing
'glance_api_insecure'. It skips the SSL validation in quantumclient.

DocImpact

Change-Id: I275517179929710bedcef5086a040b9396c0e932
Flavor Extra Specs should require admin privileges

The previous fix added admin check in policy.json,
but code still can't recorginize the detailed actions.

This fix edited "authorize" function for flavor_extra_specs,
to make sure it will check the admin privileges in policy.json.

Also, together with the code, this fix edit old test case with admin privileges,
and added new non-admin privileges test case.

Fixes bug 1094142

Change-Id: Ia286aedb4846383ad51bd54b0984dd1feddfbf81
Update modules from common required for rpc with lock detection.

This change updates the versions of the local, lockutils and rpc
modules from common. This is required so that I can implement
automated detection of rpcs being made while holding locks, which
will come in a later review.

Sneaks up on bug 1063222.

Change-Id: I3689c133d1efc7425a44662abbdb434b5f36c64e
Extension for rebuild-for-ha

Adds an extension for blueprint rebuild-for-ha

Evacuation of VM from a failed host for HA.
New api supports recreate/rebuild modes. With shared storage,
VM is recreated on the target from disk.
Otherwise, rebuild from original image is performed.

DocImpact

Change-Id: Id7e3e68dbaaf74e9314440d98195a92454078fd9
Co-authored-by: Oshrit Feder <oshritf@il.ibm.com>
Merge LibvirtHybridOVSBridgeDriver into LibvirtGenericVIFDriver

The LibvirtHybridOVSBridgeDriver VIF driver and the
LibvirtOpenVswitchDriver VIF driver both support OpenVswitch
based guest configs. The difference is that the hybrid driver
does not connect the VM directly to the OVS bridge. Instead it
connects the guest to a private Linux bridge, connected to the
OVS bridge via a veth pair. This allows iptables firewall rules
to be applied to the guest VIF. The hybrid driver should always
be used unless the admin has disabled Nova firewalls using the
NoopFirewallDriver impl. Rather than require the admin to figure
this out themselves, pick the right impl automatically based
on the configured firewall driver.

When Quantum gains support for firewalling, it will need to be
able to inform Nova whether it has got a filtering applied to
the VIFs, and thus let Nova intelligently skip its own attempt
at filtering.

The LibvirtGenericVIFDriver class can use the 'vif_type' mapping
field to determine whether an OVS network configuration is
required. In combination with a check against the firewall
driver, a dedicated driver for OVS hybrid setup is no longer
required.

The LibvirtHybridOVSBridgeDriver class functionality is
merged into LibvirtGenericVIFDriver. For backwards compatibility
with the Folsom release, the existing LibvirtHybridOVSBridgeDriver
class is made to inherit from LibvirtGenericVIFDriver and directly
call the bridge specific setup APIs. This eases migration to
the new VIF impl during the Grizzly deployment lifecycle, with
an expectation that the LibvirtHybridOVSBridgeDriver stub
will be deleted after the Havana release.

Bug: 1050433
Blueprint: libvirt-vif-driver
Change-Id: I93294409a75aaa15d297471ab3e600c3c124bcde
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Introduce support for 802.1qbg and 802.1qbh to Nova VIF model

Define the nova network VIF model properties for supporting
802.1qbg / 802.1qbh networking in hypervisor drivers. Also
wire up support to the libvirt driver for these new modes

Blueprint: libvirt-vif-driver
Change-Id: I91039790900170bfd698ea484d5f8a22445806ae
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
xenapi: Cleanup detach_volume code

Refactors `detach_volume` so that it uses the same code to detach that
detaching all volumes uses.

Change-Id: Idb7acb136367d8ad512d4b14c775c69404f6f6b0
Change to support custom nw filters.

This change is a refactoring of NWFilterFirewall to
create a distinct function to define base nw filters.
The new function is called get_base_filter_list.  This
provides a convenient way to subclass NWFilterFirewall
and redefine get_base_filter_list to add your own custom
base filters.

Change-Id: Ida94148fcb870830ae675839ab4f900ad798d9bb
Fixes: bug #1103086
support reloctable venv roots in testing framework

run_tests.sh, with_venv.sh, and the install_venv.py scripts now
support relocating the venv root to another location.

All 3 scripts now support new envinroment variables to configure
the location of the venv and the tools directory. To maintain
compatability the defaults are set to the current values.

venv_path = Location of the virtualenv directory
	Default: $(pwd)

venv_name = Name of the virtualenv directory
	Default: .venv

tools_path = Location of the tools directory
	Default: $(pwd)

Additionally the run_tests.sh script also takes these value as
arguments and will pass them along accordingly.

--virtual-env-path <path>   Location of the virtualenv directory
                             Default: $(pwd)
--virtual-env-name <name>   Name of the virtualenv directory
                             Default: .venv
--tools-path <dir>          Location of the tools directory
                             Default: $(pwd)

DocImpact

Change-Id: I1be036058227206ecca342f692cd3d6aadb24069
Fixes: bug #1116942
Add a virtual PowerDriver for Baremetal testing

This virtual_power_driver provides the ability for baremetal driver to
start/stop/reboot VMs within common dev/test environments. Its goal is to
approximate the same functionality as other PowerDrivers (eg. IPMI) within
a desktop development environments and within the devstack-gate environment.

Authored-by: Chris Krelle <nobodycam@gmail.com>
Co-authored-by: Devananda van der Veen <devananda.vdv@gmail.com>

Change-Id: I9935a9a30a166a9c28a9abb139c49f219a32ddd7
Fixed FlavorAccess serializer

Fixed flavor_access template
Renamed remaining serializers
Fixed serializer tests
Fixes bug 1115880

Change-Id: I28913f1466ff6d1f7933c292ba617be2899c09a7
Remove nova.db call from baremetal PXE driver

Baremetal PXE driver was fetching instance_type from the nova db.
Now that no-db-compute is done, that's not possible. Also, there's a
more efficient way -- just call extract_instance_type(instance)!

Baremetal unit tests should check for instance_types with swap = 0 and
with swap > 0. This required changing test/utils to include a
real copy of instance_types in instance['system_metadata'].

Change-Id: I2a91e5c026782946f6b01e4189ec85d30ba87583
l3.py,add_floating_ip: setup NAT before binding

fix for bug 1100435

Change-Id: Iad022f61297fe26edb230ba7b9e31d73df99b5a5
Use joined version of db.api calls

Replace db.instance_get_active_by_window with
db.instance_get_active_by_window_joined

Remove db.instance_get_active_by_window as it is now unused

When moving over to nova.db.api not returning sqlalchemy
objects non-joined db calls (lazy loaded) won't work.

Partially implements bp db-api-cleanup

Change-Id: Iec647792a049d53862a45f95accd7f46e483aa17
Update nova/compute/api to handle instance as dict

When nova.dp.api returns primitives instead of sqlalchemy objects
instance objects will be dictionaries, which cannot be added to sets
('dict' is an unhashable type)

Partially implements bp db-api-cleanup

Change-Id: I3683d1c54ae627d9cabfd569a7fd304c47c5d5b7