PXE driver should rmtree directories it created

Baremetal PXE driver was failing to delete the per-instance tftpboot and
image directories which it created when the instance was deleted. This
happened partly because of dangling files within the directory, and
partly because 'unlink' does not remove directories and the error was
squelched.

Now, when destroy()ing an instance, PXE driver will call shutil.rmtree()
on the per-instance directories which it created.

Fixes bug 1101048.

Change-Id: I682d0264288add488ea23e5d5200357b7be52dd9
Baremetal/utils should not log certain exceptions

unlink_without_raise was logging exceptions when it failed to unlink a
file, which is confusing since it is often called on non-existing files.

create_link_without_raise was also logging exceptions when it failed
to create a symlink.

This patch corrects this behaviour; both functions now explicitly check
for the type of error they expect, and then suppress that. If another
type of OSError is encountered, a warning is logged. This patch also
adds unit tests for both functions.

fixes bug 1097931.

Change-Id: Ie36f59dce34a6c67765770c5f968c003003acc88
ensure failure to inject user files results in startup error

This was the case in Essex but was inadvertantly changed in:
folsom-2-95-g0d166ca.

* nova/virt/disk/api.py: Refactor to allow specifying
mandatory injection items, that result in an exception
on failure to inject.
* nova/virt/libvirt/driver.py: Specify that user 'files'
are mandatory items and thus result in VM startup failure
unless injected successfully.
* nova/tests/test_virt_disk.py: A new test for the
separate warning and error cases.

Fixes bug: 1095744
Change-Id: Idab5c4294c1cb52098ce44a7aae957a44fb2674f
Add version constraint for coverage

Python coverage lower than 3.6 causes issues when running the whole
test suite. Some arguments are randomly mixed up with script names,
resulting in unittest failures like this:

UnexpectedMethodCallError: Unexpected method call.  unexpected:-  expected:+
- exists.__call__('/tmp/buildd/.../nova/virt/images.py') -> None
+ exists.__call__('/some/path') -> True

Change-Id: Iadb6d046e3acc75c6ed5abffcfc1abda82aeee4f
Added sample tests to floating-ip-pools API.

Partially implements blueprint nova-api-samples
Fixes bug 1070165

Change-Id: Ia7d99354ae815cdd8a41b012f568877751a3d8bd
Fixes task_log_get and task_log_get_all signatures

Fixes signatures for the above methods in nova/db/api.py
based on the recent TaskLog refactor on sqlalchemy/api.py.
Also adds unit tests for TaskLog.

Fixes Bug 1102652.

Change-Id: Id7ce74b8941d9eb6a50dfbfaa62e0fe05dd467c2
Avoid db lookup in info_from_instance().

The method info_from_instance() in nova.notifications was doing a db
lookup for the system_metadata for an instance.  This patch updates that
code to get that data from the instance that's passed in instead.  The
rest of the patch are related changes to make that happen.

metadata_to_dict() was needed here.  It lived in nova.compute.utils.
nova.compute.utils already imported nova.notifications, so using it from
there would have created a circular import.  Move the method to
nova.utils instead and update the tree to use it from its new location.

I also noticed that the xen driver had a copy of metdata_to_dict().
This patch removes it and uses the common implementation in nova.utils.

'system_metadata' was added to _extra_keys of the Instance db model so
that it would show up in a serialized instance.  Tests failed without it
as the result of getting instances via the conductor API did not include
system_metadata.  Now it's there.

Part of bp no-db-compute.

Change-Id: I451355fb26ae29f13b71438f7896c448b59f97b0
Merge LibvirtBridgeDriver class into LibvirtGenericVIFDriver

This introduces the LibvirtGenericVIFDriver class which uses
the 'vif_type' mapping field to determine what type of host
network configuration is required.

The LibvirtBridgeDriver class functionality is merged into
LibvirtGenericVIFDriver. For backwards compatibility with
the Folsom release, the existing LibvirtBridgeDriver class
is made to inherit from LibvirtGenericVIFDriver and directly
call the bridge specific setup APIs. This eases migration to
the new VIF impl during the Grizzly deployment lifecycle, with
an expectation that the LibvirtBridgeDriver stub will be
deleted in the Hxxxxx release.

DocImpact
Blueprint: libvirt-vif-driver
Change-Id: I948158b5ee0a97f666bb4d21020b552d3475b6d3
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Fix bad imports that cause nova-novncproxy to fail

Looks like it was introduced with
Iddba07ff13e10dc41a6930749044bb8c0572d279

Fixes LP# 1104907

Change-Id: Ie0c01af62a7fd20a6021b3c6fe89af88585e2fb2
clean up api_samples documentation

the documentation for API samples was very misleading,
and hard to point people at what to do to add new samples.
Bring this more in line with reality.

Change-Id: I52da343d317cc77e5df30815f3f3721c2634585b
Enable N302: Import modules only

Fix all N302 issues, and re-enable.

Change-Id: Ic94d144c915b228b7ff2fd9c5951875e159ffcdd
Merge "Added sample tests to floating-ip-pools API."
Merge "clean up api_samples documentation"
Add system_metadata to db.instance_get_active_by_window_joined

Without this compute/utils/notify_usage_exists() throws an
exception when looking for system_metadata in the instance data

Change-Id: I3606aa18b1aa69f98de820840c080f848af56157
Move policy checks to calling side of rpc.

This prepares to move some of the functionality in the manager
up to the api. We previously did policy checking on the manager
side because quantum v1 was a manager. Now that quantum replaces
the api, we can do the policy checking on the api side.

Change-Id: Id404c66e61044b880dbfb11f7cbba20cbab4b527
Merge "Enable N302: Import modules only"
Merge "Avoid db lookup in info_from_instance()."
VMware Compute Driver Networking

blueprint vmware-compute-driver

VLAN & Quantum integration
DocImpact

Nova conf:
[vmware]
integration_bridge=Name of integration bridge, defaults to 'br-int'

Change-Id: Ib56b8a49773b933be82de04576deb832ef89a0bd
VMware Compute Driver Host Ops

blueprint vmware-compute-driver

Host power action and maintenance mode
Get host stats and update host status

Change-Id: I88eff5482ad50a8ffa262dd015d2395fe2095fba
VMware Compute Driver OVF Support

blueprint vmware-compute-driver

Launch OVF disk image
DocImpact
Add support for VMware disk and adapter type

vmware_disktype: eagerZeroedThick, preallocated and sparse
Default value is 'preallocated'.
'sparse' disk refers to OVF supported VMDK format
(sparse or streamOptimized).
A sparse disk is copied to thin disk upon deployment.

vmware_adaptertype: ide, busLogic and lsiLogic
Default value is 'lsiLogic'.

Usage:
glance add name="Ubuntu" disk_format=vmdk container_format=ovf
is_public=true vmware_adaptertype="lsiLogic"
vmware_ostype="ubuntuGuest" vmware_disktype="sparse"
 < ubuntu-disk1.vmdk

Add support for VNC console

Nova conf:
vnc_enabled=[true|false] defaults to true
vnc_port=[port #] defaults to 5900
vnc_port_total=[total port #] defaults to 10000,
the actual VM VNC port is vnc_port + VM ID % vnc_port_total
vnc_password=[optional]

Change-Id: I912b46b9f72f9089b448e03a04327fb79b9f0f3f