Enforce metadata string type on key/value pairs

The compute API is currently enforcing quota limits for user metadata
and length restrictions on the dict entries, but is not enforcing that
they are string types. This patch adds the checks for string type on the
keys and values in the metadata dict.

Closes-Bug: #1240319

Change-Id: Ic4aacc342cf37d929871287e6e4917f4f48b63e8
Fix LibvirtGenericVIFDriver.get_config() for quota

When Nova is used with Neutron and Open vSwitch, network quotas on the
bandwidth are ignored by libvirt: the "<bandwidth>" tag is not generated in
libvirt.xml of the virtual machine. This change fixes this issue but also for
the network types: "bridge", "network" and "direct".

Commands to create the flavor with download limited to 1 MB/sec and create a
virtual machine with this flavor:
---
nova flavor-create --ephemeral=1 victor_test_vif 50 256 1 1
nova flavor-key victor_test_vif set quota:vif_inbound_average=1000
nova boot --flavor=victor_test_vif --image=cirros-0.3.1-x86_64-uec victor_test
---

Commands to upload a file to this virtual machine with SCP to check that the
bandwidth is limited to 1 MB/sec (replace 10.0.0.3 with the IP of the virtual
machine):
---
ip netns
sudo ip netns exec qrouter-128db593-a0db-40c3-84c7-e6383d40c75f bash

dd if=/dev/urandom of=random10MB bs=1024 count=10240
scp random10M cirros@10.0.3:
---

Change-Id: I0f0111fc79fe90900e38df022034b208f1129088
Closes-Bug: #1254664
DB migration 209: Clean up child rows as well

Fixes bug 1252891

In order to add missing foreign key constraints, rows that reference a
nonexistant key need to be purged. Unfortunately, DB migration 209 does
not clean up any rows in child tables that themselves have a foreign key
constraint on the tables with the missing foreign keys. This can result
in an integrity violation causing the migration to fail.

When purging the rows, make sure to purge rows in any child tables first.

Change-Id: I25bd899383c72cb0670819b030759630ac89acaa
Update Instance from database after destroy

In cases where Nova falls back to simply locally deleting an
instance, the Instance object was not updated with the deleted_at
generated by the database implementation. Due to this, the
notification emitted to signify that the delete finished was
getting emitted without a deleted_at value.

This change uses the result from the instance_destroy call to
update the Instance object with the latest state from the database.
Thus, the compute.instance.delete.end notification is emitted
with the proper deleted_at value.

Closes-Bug: #1250206
Change-Id: I55ed37cf89dd6eb8ea901c6f813191a0ce1058e8
Add shutdown option to cleanup running periodic

Add a 'shutdown' option to the periodic task that cleans up instances
which are both deleted *and* running.

The option powers down instances and marks them as non-bootable.  This
is a flag for system administrators who would like to reclaim the
resources consumed by each instance, but leave the records around for
later examination/debugging.

DocImpact
Implements: blueprint cleanup-running-deleted-shutdown

Change-Id: Id95845eb995b62ed8858fe61c6fdb34045721674
Log exception in _heal_instance_info_cache

Previously, if errors occurred in _heal_instance_info_cache() they would
go unnoticed as their log level was debug and the stace trace was not
displayed. This patch changes that so the exception is logged so periodic
failures here can be more easily detected and fixed.

Change-Id: I15bb3e7e29462e930f6159be9f4bbf4980b1270c
Closes-bug: #1257940
Move calls to os.path.exists() in libvirt imagebackend

The checks for the path existence were removed from the base Image class
to avoid redundant checks (some of implementations also perform them and
behave differently depending on the results). Also special handling for
some backends were removed from the base Image class.

Change-Id: Ie6cdf0d6d9801f84fae7a58c1034547611712323
Closes-Bug: #1152748
Replaces call to lvs with blockdev.

Replaces the call to lvs in virt/libvirt/utils.py:logical_volume_size()
with the more general blockdev.

Implements: blueprint encrypt-ephemeral-storage
Change-Id: I3e4e4e927a98fa9e396618e45ac01cc836a75988
Use a dictionary to eliminate the inner loop in _choose_host_filters()

it looks like an o(n²) algorithm, we can use a dictionary to eliminate
the inner loop.

Change-Id: I32747120ae1f2f1bc0a3810a03e0cd9baa7a8d2c
XenAPI: Synchronize on all VBD plug/unplug per VM

Dynamically adding or removing a device needs to be synchronized
to work around a bug in XenServer 6.1 and 6.2 where simultaneous
events can be partially ignored, leading to a failure to plug or
unplug.  The bug is between two internal components in XenServer
which pass messages - the first component will ignore all messages
from the second while a plug/unplug is in progress, expecting
only one message to be received.  If two plug/unplugs occur in
this time, an internal assertion is raised as XAPIs internal state
is inconsistent.

Change I7b86325714e38b00a2f51bbf083b5532a6c51fa9 attempted to
work around the issue by retrying - however that does not catch
all failures.  This change should eliminate some of the known
INTERNAL_ERROR failures, but not all, so I7b8 need not be
reverted.

Change Ie7a64c481a9e3874df9d5924fbb050ac7ab117db does the same
as this, but only for VBD.plug.

Change-Id: If78d4e5f1dd84f1d4a588f25b8f887a7b125bbea
Closes-bug: 1217972
Don't replace instance.info_cache on each save

If we are working on an Instance that came from an older client,
we should not replace an older nested .info_cache object with a
newer one for no reason during a save().

Related-Bug: #1258256
Change-Id: I8cf4a629d454de1a5de16ea4032f39e36aab505a
Merge "Move calls to os.path.exists() in libvirt imagebackend"
Merge "Enforce metadata string type on key/value pairs"
Merge "Add shutdown option to cleanup running periodic"
Merge "Replaces call to lvs with blockdev."
Merge "Use a dictionary to eliminate the inner loop in _choose_host_filters()"
Merge "XenAPI: Synchronize on all VBD plug/unplug per VM"
Merge "Don't replace instance.info_cache on each save"
Merge "Update Instance from database after destroy"
Merge "Log exception in _heal_instance_info_cache"