ensure zeros are written out when clearing volumes

Note O_DIRECT is _not_ used at the end of devices to
avoid issues with odd sized blocks etc. so instead
we arrange to have dd issue an fdatasync()
to ensure the data is persisted, lest it be discarded
from the write cache when the device is unprovisioned.

* nova/virt/libvirt/utils.py (clear_logical_volume): Add 'conv=fdatasync'
to the dd option list if O_DIRECT isn't used when clearing.

Fixes bug: 1100363
Change-Id: I76789557754ebaeb6d52bb34548a2ef17808fbf6
Cells: Add support for compute HostAPI()

This will allow the hosts API extension to work. Calls to get service
information from the DB are proxied via cells.

When using the host extension, the 'id' and 'host' fields for
service/compute will be modified to have the cell prepended to it.  One
can do a 'show' on a 'path!to!cell@id' to see details for a particular
host in the 'path!to!cell' cell, for instance.

Implements bp/nova-compute-cells

Change-Id: I8137fa64d2578acfc3d2457c231de2b0d69b96d8
Update en_US message translations.

Fixes bug 1101587.

* nova/locale/en_US/LC_MESSAGES/nova.po: This "translation" was
woefully outdated, causing KeyError exceptions under a number of
circumstances for users of this particular locale when format
strings wound up containing incorrect substitution variables. This
change attempts to sync up the msgstr entries to match msgid given
this is the source language for the pot file. I did not see any
obvious instances where en_US should differ from C locale for these.

Change-Id: I81d1729fee17653b089f3c47db20f3fc073ab363
Add support for memory overcommit in live-migration

Override the live-migration memory check in FilterScheduler and
make it use RamFilter for checking available memory on target
host.

Resolves bug 1068258.

Change-Id: I3002b8330e595ce71e2718b2091d3b0b99747707
VMware Compute Driver Glance improvement

blueprint vmware-compute-driver

Glance change to fix image download to ESX
The current glance code downloads files in a blocking/synchronous
manner, the fix improves it to work with ESX glance file reader
in a non-blocking manner.
This allows ESX glance file reader to read items from the returned
queue.

Change-Id: I05ab7ac00878aeeea3bbcd7546b05398f2630dd3
populate dnsmasq lease db with valid leases

At start dnsmasq retrieves its lease db by calling the nova-dhcpbridge
script with the init action.

Previously we were returning all allocated IPs in the network and
calculating the lease exirpation time based upon instances.updated_at
or instances.created_at.

Now we are only returning entries for IPs that are allocated and leased.
Additionally we set the lease expiration time to be i
now + CONF.dhcp_lease_time

Change-Id: Ie3b4fafd38094dc01ceea592ab5229e581193512
Fixes: bug #1103260
Add api-samples to multinic extension

Partially implements bp nova-api-samples
Fixes bug 1070169

Change-Id: I923054bf92840a7fe81f78c97aa77a9201e94bdc
Relax API restrictions around the use of reboot

Fixes bug 1101082 bug 982108

Reboots are the only sledge hammer users have available to resolve
issues with their instances without administrative intervention.
This patch modifies the API policy to allow reboot calls to be made
in many more power and task states.

Change-Id: Ia8702448f6b7b863da40e4d498f2e2ee0a12882e
Refactoring/cleanup of compute and db apis

Refactor db call usage and remove some unused code.

Change-Id: I49466f7ab54dd6dbf0fc78bab3d2df44392ba3ee
fix nova-baremetal-manage version printing

version.version_string_with_vcs is no longer present

Change-Id: Iaecf793b82c030f7095de7e42222e28869d1cf33
Removes retry of set_admin_password

* An RPC timeout may occur if an agent is missing and set_admin_password is
invoked. This causes 500 errors in the OpenStack API.
* Implemented a 501 error in API if the password set fails.
* Modified xenapi agent to use NotImplementedError instead of Exception in
  set_admin_password.
* Updated test code around set_admin_password to accept different exceptions.
* Fixes bug 1061045

Change-Id: If7fab56c20f12e0490f4774e00004ed1d94242b9
Pass parameter 'filter' back to model layer

This allows the API to pass back additional arguments to filter search
results by.  This will be useful for resource tagging or other filters.

DocImpact: Adds 'Filter' parameter to DescribeInstance (see
http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiReference-query-DescribeInstances.html)

Fixes: bug #922835

Change-Id: I58cc12df650829ed7eb685184a22186d8a5406f4
Signed-off-by: Stephen Gran <stephen.gran@guardian.co.uk>
Add option to control where bridges forward

This adds a new config option to control where bridges can forward.
The default behavior is unchanged, but if you specify:

forward_bridge_interface=foo

Then only traffic on the same bridge or going to and from the foo
interface will be allowed. This is useful in vlan mode when a
deployer doesn't want traffic to be routed between vlans. Note
that forward_bridge_interface can be specified multiple times and
should be specified once for each interface that supports floating
ips. i.e.:

forward_bridge_interface=eth0.10
forward_bridge_interface=eth0.20

DocImpact

Change-Id: I82d6b288b60cd6deae64690b55546730b1c2da55
Add api samples to fping extension

Partially implements bp nova-api-samples
Fix bug 1091747

Change-Id: Ic0060b8ea2a01a304c337a31dc276cbf395c2e5d
Remove deprecation notice in LibvirtBridgeDriver

After further discussions with Quantum team, it was agreed
not to deprecate the existing libvirt VIF drivers until the
Havana release, to allow more time for Quantum plugins to be
updated.

Blueprint: libvirt-vif-driver
Change-Id: I0226c4117630de489543d35662a8ee6b6c7729f9
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Merge QuantumLinuxBridgeVIFDriver into LibvirtGenericVIFDriver

The only difference between the Quantum bridge driver and the
new generic VIF driver bridge support is the name of the bridge.
This is now provided via the VIF mapping parameter, so there is
no need for the dedicated Quantum VIF driver for bridging. This
This has the advantage that libvirt firewall filter parameters
are correctly configured.

The QuantumLinuxBridgeVIFDriver class functionality is merged into
LibvirtGenericVIFDriver. For backwards compatibility with
the Folsom release, the existing QuantumLinuxBridgeVIFDriver class
is made to inherit from LibvirtGenericVIFDriver and directly
call the bridge specific setup APIs. This eases migration to
the new VIF impl during the Grizzly deployment lifecycle, with
an expectation that the QuantumLinuxBridgeVIFDriver stub will be
deleted after the Havana release.

Blueprint: libvirt-vif-driver
Change-Id: I19664f20e5137f495513999214f66279292bda58
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Merge LibvirtOpenVswitchDriver class into LibvirtGenericVIFDriver

The LibvirtGenericVIFDriver class can use the 'vif_type' mapping
field to determine whether an OVS network configuration is
required, thus a dedicated driver for OVS is no longer required.

The LibvirtOpenVswitchDriver class functionality is merged into
LibvirtGenericVIFDriver. For backwards compatibility with
the Folsom release, the existing LibvirtOpenVswitchDriver class
is made to inherit from LibvirtGenericVIFDriver and directly
call the bridge specific setup APIs. This eases migration to
the new VIF impl during the Grizzly deployment lifecycle, with
an expectation that the LibvirtOpenVswitchDriver stub will be
deleted after the Havana release.

Blueprint: libvirt-vif-driver
Change-Id: Iac79e0880b6fbcca0fd8cc26b59d23e579b947fa
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Add support for compressing qcow2 snapshots

Fixes bug 1109923

Adds a new configuration option:
    libvirt_snapshot_compression=False (Default)

When set to True and the snapshot output format is qcow2, we pass
the -c option into qemu-img, which enables compression. This patch
also refactors the extract_image function to make construction of
the whole qemu-img command more dynamic.

Change-Id: I7a7ebcf41a91a8a9cb14be79e9bb79c22acb136e
Flags: DocImpact
Code cleanup for rebuild block device mapping

* Removes extra call to get block-device-mappings when detaching during
  recreate.

* Untangle network handling from block-device handling in rebuild

* DRYs up evacuate hosts tests

* Temporary mutation supports dicts as well as objs

* Make instance recreate exceptions more specific to aid testing

Change-Id: I242f9f6a7d329f27b06b7e81b9a418e01682c82e
xenapi: Ax unecessary `block_device_info` params

The xenapi driver uses the 'osvol' other_config to determine which
volumes to detach, making `block_device_info` unecessary for internal
functions.

`destroy` still needs to accept the param since it's part of the
virt-driver API.

Change-Id: I1bbd69f41f9a1e18d4759d0ef3c6de027d3c9981