Fix unicode string values missing in previous patch

Commit 61bedc9aa04b0bac21e5c4bd17d6d2c09e2d9e01 lost some code
around the str_or_none() helper in objects/utils.py during a
rebase which caused almost all of our values to not actually
be coerced to unicode. This adds that.

Also, the instance.uuid value can't (and shouldn't) be unicode,
so add a cstring() helper and convert it to use that so that
the above change doesn't try to make uuid be unicode. Also,
make str_value() enforce non-None-ness like cstring() to avoid
bugs where we get 'None' unexpectedly, and add a test for that
whole function, which was missing before.

Change-Id: I941545d38aee10f140610b268fbd9c744f6545f9
VMware: Fix SwitchNotFound error when network exists

In FlatNetworking mode, when a virtual network exists on the vCenter
host we still check for the presence of a physical adapter specified
in vlan_interface. This check is only needed if we are going to
create a new network on the vswitch associated with the interface.

This fix removes the check for the physical adapter and the vswitch
when network already exists.

Fixes: bug #1226238
Change-Id: I1eecf2c339534d8694ec6a8bdfaff5217ba2f8ef
Adds V3 API samples for srvcs, tenant usage, server_diagnostics

Adds V3 API samples for the services, simple tenant usage and
server_diagnostics extensions. Bundling them together to save
on review load

Partially implements blueprint v3-api-unittests
Partially implements blueprint v3-api-specification

Change-Id: I91edc6500acbe92221427383b4bcd3d00a8faadd
fix the an Unexpected API Error issue in flavor API

The flavors.add_flavor_access and flavors.remove_flavor_access
require admin privilege in DB level but the policy doesn't
require it and the exception isn't catched in the api. So the
bug arise. The privilege check in DB level is not necessary,
but it need more tests and can be remove in other patch or
blue-print.

This use extension_authorizer instead of soft_extension_authorizer
in the action of flavor_access, in order to raise exception.

Closes-Bug: #1217679
DocImpact

Change-Id: I0b1231f47fffaf99f330de02956bfe8d7cd4b920
get_all_flavors uses id as key to be unique

Change I1b0c93b3e12714a5c0798aa87c9e3fe9bc14519a fixed returning
deleted flavors by changing the get_all_flavors dictionary key
away from 'name' to 'flavorid'. However flavorid is also not unique.

This patch fixes get_all_flavors to work properly using the 'id' as
the dictionary key. This allows flavors to be created with flavorid's
that were previously deleted and then have them all returned when
get_all_flavor(inactive=True) is used.

Fixes bug 1215295 properly

Change-Id: I357a804fe9f012143c996d66b9719c4f46f60e86
Make sure periodic task sync_power_states continues on error

If an instance sync_power action fails with an exception, no more
instances will be synced. If the failure is permanent, this will block
other instances from having their power states synced for as long as
this keeps happening.

Make sure to catch and log such exceptions and then continue with the
next instance.

Change-Id: Ie3cf5d7f590cb680e712d76b7a32c630e82f362d
Closes-Bug: #1228160
Ensure image property not set to None in build_request_spec()

This is in preparation for having scheduler.util.build_request_spec()
construct the request_spec in the live migration code. Specifically, this
change is needed to avoid reintroducing bug #1199811.

Change-Id: I0cc6b6ec7be7a0afbe184b208f2e16fd5cd4452c
Related-Bug: #1224014
Make LiveMigrateTask use build_request_spec()

Some filters like TrustedFilter makes use of extra_specs. Currently,
when live-migration uses the scheduler to select a host, it constructs
a request_spec that has no extra_specs in it.

By making use of the existing helper method build_request_spec(), the
request to the scheduler will include extra_specs.

Change-Id: I5bc6c6418653c256a42da7b0a343086ec9863da1
Closes-Bug: #1224014
Add notification for pause/unpause instance

When pause/unpause instance, there are no notifications, this
will cause someone cannot know when pause/unpause operation
would be finished.

All other instance operations do have notifications. We should
also add notification support for those two operations.

Change-Id: I5b8cc33a5992f5f028a3220217dcf7a0288d150d
Closes-Bug: #1043861
Skip test if sqlite3 not installed

The test is attempting to run the sqlite3 binary during its test run.
Since there's no guarantee that it's installed it should catch an
execution error and skip.

Change-Id: I9f4ed1872fcfe3b8393644a77692dbfcce8ecab2
Closes-bug: 1227868
Resource limits check sometimes enforced for forced scheduling

Resource limit checks on a compute manager are based on the
contents of filter_properites['limits'].   This in turn is
populated by the scheduler after the host has been selected by
scheduler.utils.populate_filter_properties.   The limits value
is taken from the host_state, which is maintained as a local data
in the host manager. The host_state limits values are populated
by the various filters, for example the ram filter sets
limits['memory_mb'] based on its configuration.

In the case of scheduling to a forced host the filters are
bypassed, so the contents of the host_state['limits'] value
depends on whether the scheduler has already placed a non-forced
instance onto that host (and thus run the filters to populate the
limits).   If it has then the compute manager will impose the
limits, if it hasn't then the limits will be empty and there will
be no check.

As forcing bypasses the filters and is a privileged operation the
limits should always be cleared in the filter_properties in this
case to avoid the limit check on the compute manager.

Fixed bug: 1231963

Change-Id: Ic7ff5d1fc652a8bdf6a0cf8e6ef063d9f2e7ab16
xenapi: Resize operations could be faster

 * Improves the code by using only one time resize2fs during the resize.
 * Removes the not used method: "_get_min_sectors".
 * Uses a VDI clone instead of copy in resize_disk's method.
 * Review of some test cases to accept the new behavior
   without _get_min_sectors.

Closes-Bug: 1221332
Change-Id: I40a1d7d467353ba904689d128de7f89d785b4052
Forced scheduling should be logged as Audit not Debug

Forced scheduling to a node is a privileged operation which bypasses
resource checks. As such the log entries should be Audit not Debug

Fixed Bug: 1231983

Change-Id: Id2016b7bf2ba5f7af4896b5f8df481437d2ee956
xenapi: reduce impact of errors during SR.scan

There are some race conditions in SR.scan leading to errors similar to:
* The SR scan failed [opterr=['MAP_DUPLICATE_KEY', 'VDI', 'sm_config',
  '$UUID', 'vhd-blocks']]
* The SR scan failed [opterr=['UUID_INVALID', 'VDI', '$UUID']]

This seems to be happening because an SR.scan is happening when a xenapi
plugin is modifying the SR at the same time.

To reduce the likely-hood of this error, I have added a mutex around
calls to SR.scan to ensure nova-compute never makes more than one call
at once, because XenServer will join any extra SR.scan requests to
listen to the result of any current SR.scan that may be running.

In addition, when and error occurs during an SR scan, I have added a
retry loop, to attempt to stop these races from impacting users.

Fixes bug 1221293
Change-Id: Ib83e6e0f4f7fab2058fe137e32934606c4fe8f7e
Merge "xenapi: enforce filters after live-migration"
Fix log format error in lazy-load message

The just-merged patch to log lazy loads missed the format specifier
in instance.py. This adds that back.

Change-Id: I6f1641a96ccffac80e609e3d304e0f21f693f1fa
Full sync of quota_usages

Migration 203 added a user_id column to the quota_usages table.  Now
some usages are incremented/decremented on a per user basis but there
are preexisting usages stored with a NULL user, and some usages are
tracked per project but there are usages stored with a user_id.  These
quotas are never decremented leading to invalid quota limits being
enforced.

This migration performs a full sync of quotas, and removes the incorrect
rows for each resource type.

Change-Id: Iff96f81e36b135df94a6f25c9443d0e9333bd18f
Closes-Bug: #1212798
Merge "Fix unicode string values missing in previous patch"
Merge "VMware: Fix SwitchNotFound error when network exists"
Merge "Make sure periodic task sync_power_states continues on error"