libvirt: use tap for non-blockdevice images on Xen

This patch reverts 35c4962c0b97bae5b8751d316d5822fe22c1ab6a: 'use
file instead of tap for non-blockdevice images on Xen', which breaks
qcow2 disk working on Xen. If the method pick_disk_driver_name()
returns 'file' for qcow2 format, we would get the following disk XML
snippet:

    <disk type="file" device="disk">
      <driver name="file" type="qcow2" cache="none"/>
      <source file="/data/instances/instance-00000081/disk"/>
      <target bus="xen" dev="hda"/>
    </disk>

which produces configuration in the xen tools like:

    "file:/data/instances/instance-00000081/disk,hda,w"

Guest would not boot successfully, as Xen doesn't think of disk format
as qcow2. Instead, 'tap' works with both raw and qcow2 disk.

For LP#1084618, I think it is related with libvirt verion. In my
deployment(libvirt-1.0.1), 'tap' dirver could produce correct
configuration for both raw and qcow2 disk format.

Change-Id: I636b2fed366474e2ae8c3b52403e3085724b24f0
Add rootwrap filters for password injection with localfs

Allow to 'sudo cat' to read passwd and shadow.

bug 1098077

Change-Id: Ic734bd33223df879b5e1f144bb4c85702eb88dfa
Fix floating ips with external gateway

If dnsmasq is configured to use an external gateway, then floating
ips on other interfaces do not work properly. This is because
outgoing traffic is no longer snatted to the floating ip.

This patch fixes it by adding an ebtables rule to force traffic
from ips that have a floating ip associated to route instead
of bridge.

Fixes bug 1096985

Change-Id: I8e4904660d42fe51c44b66686bed9f5d622693bd
Add version constraint for cinder

Cinder client usage requires the "retries" parameter introduced in
cinderclient-1.0.1. This should be a constraint in pip-requires.

Change-Id: I36e8a363c9d4bb7cc4ecfda85644fc4284f17471
Add version constraint for coverage

Python coverage lower than 3.6 causes issues when running the whole
test suite. Some arguments are randomly mixed up with script names,
resulting in unittest failures like this:

UnexpectedMethodCallError: Unexpected method call.  unexpected:-  expected:+
- exists.__call__('/tmp/buildd/.../nova/virt/images.py') -> None
+ exists.__call__('/some/path') -> True

Change-Id: Iadb6d046e3acc75c6ed5abffcfc1abda82aeee4f
_sync_compute_node should log host and nodename

ResourceTracker:_sync_compute_node logs the host name when it
creates or updates the record for a compute node. However, with
baremetal, one compute_node (host) may actually manage many
separate physical nodes, identified by unique nodenames.

This patch updates the LOG.info calls to include the nodename.

Change-Id: I69c66b84394b2a740a7ca9d2781015a7e1ff9998
Fix rendering of FixedIpNotFoundForNetworkHost

Fixes bug 1104000

With https://review.openstack.org/18319 a small bug was introduced. The
exception FixedIpNotFoundForNetworkHost is expecting network_id
parameter, and the change modified that parameter to network_uuid. This
patch fixes this problem at the raise point.

Change-Id: Idc0fa7fcba9ad37b73183f8811376c6c8adada68
Merge LibvirtBridgeDriver class into LibvirtGenericVIFDriver

This introduces the LibvirtGenericVIFDriver class which uses
the 'vif_type' mapping field to determine what type of host
network configuration is required.

The LibvirtBridgeDriver class functionality is merged into
LibvirtGenericVIFDriver. For backwards compatibility with
the Folsom release, the existing LibvirtBridgeDriver class
is made to inherit from LibvirtGenericVIFDriver and directly
call the bridge specific setup APIs. This eases migration to
the new VIF impl during the Grizzly deployment lifecycle, with
an expectation that the LibvirtBridgeDriver stub will be
deleted in the Hxxxxx release.

DocImpact
Blueprint: libvirt-vif-driver
Change-Id: I948158b5ee0a97f666bb4d21020b552d3475b6d3
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Add debug log when call out to glance

Calling out to another service may take a while, so lets log this

Change-Id: I223105ee578ae82230cbc3936c096e148b07e8f1
Move policy checks to calling side of rpc.

This prepares to move some of the functionality in the manager
up to the api. We previously did policy checking on the manager
side because quantum v1 was a manager. Now that quantum replaces
the api, we can do the policy checking on the api side.

Change-Id: Id404c66e61044b880dbfb11f7cbba20cbab4b527
VMware Compute Driver Networking

blueprint vmware-compute-driver

VLAN & Quantum integration
DocImpact

Nova conf:
[vmware]
integration_bridge=Name of integration bridge, defaults to 'br-int'

Change-Id: Ib56b8a49773b933be82de04576deb832ef89a0bd
VMware Compute Driver Host Ops

blueprint vmware-compute-driver

Host power action and maintenance mode
Get host stats and update host status

Change-Id: I88eff5482ad50a8ffa262dd015d2395fe2095fba
VMware Compute Driver OVF Support

blueprint vmware-compute-driver

Launch OVF disk image
DocImpact
Add support for VMware disk and adapter type

vmware_disktype: eagerZeroedThick, preallocated and sparse
Default value is 'preallocated'.
'sparse' disk refers to OVF supported VMDK format
(sparse or streamOptimized).
A sparse disk is copied to thin disk upon deployment.

vmware_adaptertype: ide, busLogic and lsiLogic
Default value is 'lsiLogic'.

Usage:
glance add name="Ubuntu" disk_format=vmdk container_format=ovf
is_public=true vmware_adaptertype="lsiLogic"
vmware_ostype="ubuntuGuest" vmware_disktype="sparse"
 < ubuntu-disk1.vmdk

Add support for VNC console

Nova conf:
vnc_enabled=[true|false] defaults to true
vnc_port=[port #] defaults to 5900
vnc_port_total=[total port #] defaults to 10000,
the actual VM VNC port is vnc_port + VM ID % vnc_port_total
vnc_password=[optional]

Change-Id: I912b46b9f72f9089b448e03a04327fb79b9f0f3f
Strip out Traceback from HTTP response

We should not let stack traces leak to the REST API users

Fixes LP# 1103324
Fixes LP# 1092610

Change-Id: Ic1208bfeb7d44e672f8b8d3a4004a9802c14e7b9
Don't warn up front about libvirt loading issues in NWFilterFirewall

Use the same technique as in libvirt/driver.py to delay import of
libvirt module

Change-Id: I500ddde4651f0f747c54533a1a308eee5a87f717
iptables-restore error when table not loaded.

When adding openstack rules, if a table module is not loaded,
the resulted file doesn't include the section for the missing
table, making new rules added to it, out of place and
a no valid file for iptables-restore

Fixes bug #1103436

Change-Id: I34ae51a23efec57bfec37b8fa378d043fcf62d70
Avoid hard dependency on python-coverage.

Updates the coverage extension in the OpenStack Compute API to
avoid a hard dependency on the python coverage module. We now
try to import 'coverage' and if it isn't present we return
an HTTP 503 (service unavailable error).

This gets rid of WARNING errors which occur when running Nova API
when the 'coverage' module is not available:

2013-01-27 17:10:28.938 16018 WARNING nova.api.openstack.compute.contrib
[-] Failed to load extension
nova.api.openstack.compute.contrib.coverage_ext.Coverage_ext: No module
named coverage

Fixes LP Bug #1101057.

Change-Id: I573180aead44d68b8377cd66f30fc6ac849d0272
Allow users to specify a tmp location via config

Fixes bug 981104

Although the temporary directory used can be controlled via
environment variables, this patch provides a way to define it
explicitly via a config option.

The default value is None, which behaves per the doc below:
http://docs.python.org/2/library/tempfile.html#tempfile.tempdir

Flags: DocImpact
Change-Id: I47b6d8bac734f453c80d541b46086a30e847c859
Fix a typo of log message in _poll_unconfirmed_resizes

That is just a negligible typo issue, so I have not filed a bug on LP.
The typo log message is like "notRESIZED/None", and should be
"not RESIZED/None".

Change-Id: I6137b477185419b1e638be3ed5310644aff96d76
Fix an error in affinity filters

Fix a call to compute_api.get_all to properly exclude deleted instances.
Add tests to prohibit this error in the future.

After some thought, this can be further optimized to let the instance
query do all the filtering/matching.

Resolves bug 1107156.

Change-Id: I6d6a6ba44d38d363489853d0407ad4cc94046656