Return Customer's Quota Usage through Admin API

DocImpact: This patch adds an extension 'UsedLimitsForAdmin'. If this
extention is enabled then it extends the used limits API behavior, such
that admin can fetch the details of any customer's quota usage by passing
the customer's tenant id in query parameters.The API signature for the
same is 'v2/{tenant_id}/limits?tenant_id={customer_tenant_id}'

Change-Id: I89b8b5083e46b899458407426c89a3865e960faa
Implements: blueprint customer-quota-through-admin-api
Add missing conversion specifier to ServiceGroupUnavailable

The message in the ServiceGroupUnavailable exception was missing the 's'
conversion specifier.

Fix bug 1183733

Change-Id: I54869431511a72c49ccf61f5e7b1a30607959423
Removing misleading error message

This function (_get_instances_on_driver) was reporting instances are on the
hypervisor and not in the database with no consideration for the filters being
used in the database query. This resulted in error log messages indicating running
instances were not in the database when in fact the query was filtering for deleted
instances.

To avoid misleading error logs of this nature I've removed the logging from the
function. Currently the locations which call this function provide more accurate
logging on the state of the instances that are returned based on the filters they
have called _get_instances_on_driver with.

bug 1183276

Change-Id: I4ee52fe417f73a90cf157bc531b483be93e34275
libvirt: improve the specification of network disks

Allow zero or more hosts, and make the name optional, since only a
host is needed to use nbd. This allows cinder to fully specify the
location of a sheepdog or rbd volume without depending on extra
configuration files, and lays the groundwork for supporting nbd.

Rename the internal attribute source_host to source_name to reflect
its usage in the libvirt xml, and add source_hosts and source_ports
attributes to store the new information.

Fixes: bug 1077817
blueprint better-libvirt-network-volume-support
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

Change-Id: I8ac431751692e52ba0786768cea996388962922d
Normalize path for finding api_samples dir

When nova/tests/integrated and doc/api_samples are installed
in different subtrees and are only combined via symlinks,
running test_api_sample fails due to traversing into the
wrong tree. Use path normalization to avoid this problem.

Fixes: LP Bug #1186775

Change-Id: Ibe9e2442dc9bf3ba7bcb429cbac124dcaeea2cef
Silence exceptions from qpid connection.close() (from oslo)

This is commit 15d8d698b7 in oslo-incubator.

Fixes bug 1185465

Don't log exceptions that happen during connection.close().  They're
not surprising (a closed socket can't be read or written), don't
require sysadmin attention, and provide red herrings in the logs
to distract from finding real root causes of problems.

Change-Id: Ia0e25fbc11094b5ad5d6d2cf7d0e786acb3df1f6
correctly set iface-id in vmware driver

bug 1183452

Commit 70c659059b8fbef811ce79700aecb01c60242ebd updated the vmware
driver to use the new vif model, but was incorrect in how it tried to
grab the iface-id and set it in the vmx file spec.  This patch fixes
that issue and adds a check for this in the existing unit tests.

Change-Id: I756539871d1844a828f9a0a295fc9fa3e59610f7
Allocate networks in the background

Setting up networks and allocating IP addresses has the potential to
take an undesirable amount of time, blocking the build of a new
instance.  We can parallelize some work by querying for this information
*while* a new instance is being provisioned in the virt driver.

This adds a wrapper around the NetworkInfo model which allows one to
query for network information in a greenthread.  When a code path is hit
that actually needs the network information, it waits for the
greenthread to finish, if it hasn't already.

Network allocation will only be async for virt drivers that accept this
NetworkInfo model.  Other drivers that expect the legacy format end up
waiting for the operation to complete before driver.spawn is called.

Implements blueprint async-network-alloc

Change-Id: I389f939b9a36bc20b3b66e98d006d694622bd963
Regenerate missing resized backing files

Fixes bug 1185588

The only way to reliably determine whether we are using a legacy
(pre-Grizzly) backing structure is to look at the backing file
reference for the instance/disk itself. Thus, if we determine the
backing file contains an underscore, we use the value following it
as our legacy_backing_size, which is then used to create the
necessary backing file.

Change-Id: I145f71068c972a604e5361ae79d6315286136786
Converts 'image' to json primitive on compute.rpcapi.prep_resize

The 'image' should be converted to primitive since it contains
complex structures. Today it works because it relies on
scheduler.rpcapi.prep_resize which does the conversion.

This is part of other changes that will move resize/cold migration
to conductor. scheduler.rpcapi.prep_resize will be moved and then
compute.rpcapi.prep_resize has to properly convert parameters.

Partially implements bp cold-migrations-to-conductor

Change-Id: Id89896bfb379ab9eb8bf13c3513b9290d88408fd
Refactors scheduler.chance.select_hosts to raise NoValidHost

This change is intended to refactor the method to be consistent
with the filter_scheduler.select_hosts implementation.

This method is called by scheduler.manager and it is exposed
through the scheduler.rpcapi. Thus, both implementations should
behave the same.

This change is needed by others that will move cold migration
to conductor.

Partially implements bp cold-migrations-to-conductor

Change-Id: I0f6fef43324cf829cdb26633874c537572423f77
Keypair API test cleanup

This is a cleanup in preparation for adding more
test coverage around keypairs. This patch does the
following:

* Moves keypair tests into their own file

* Breaks tests for methods into logically separate
  classes

* Shares key name validation and quota checking
  tests between `create` and `import_key`

* As part of above, add missing tests for
  `import_key` that were present for
  `create`

Change-Id: I8d8fdf9a1f9f11e1015ebbdfde1c8e77f5f421a8
Moving more tests to appropriate locations

Change-Id: I06aa8d8f31593cfea6e26560d6af2caf6a29194d
Merge "Moving more tests to appropriate locations"
Fix Keypair exception messages

`InvalidKeyPair` exceptions weren't raising the
proper error message. The reason was that the
error message was passed in as an `explanation`
keyword which isn't understood by `NovaException`,
and so is thrown away.

This is part of a patch set to improve keypair
error message reporting in both the Openstack and
EC2 APIs.

References bug 1187952
Change-Id: Ide319cb126955927d623972dffd573cd4c4314df
Improve Keypair error messages in osapi

The KeypairAPI code provides useful error messages that weren't being
percolated back through the OpenStack API. This patch addresses this by using
the original `message` when crafting the HTTPException objects.

References bug 1187952

Change-Id: I786a225010d912d7bfea5c01838ddcf43fd90d53
More KeypairAPI cleanups

* DRY up quota checking for new keys

* DRY up formatting of get_key/get_all_keys results

* Add tests for get all keypairs

Change-Id: I14a97c0f3cb3aa9b827d14002c21076a41942023
Call virt.driver.destroy before deallocating network.

Call virt driver destroy before releasing the instance's
ip address, so that certain failures can be handled.
For instance, if destroy raises InstancePowerOffFailure,
do not deallocate network resources.

Fixes bug 1180178

Change-Id: I9bde984d2393bd73a73eae6065defb2ed7c01492
Adds check that the core V3 API is loaded

Adds infrastructure to be able to check that the core V3 API
plugins are present and raise an exception and abort if any are missing. The
list is deliberately hard coded though it is not yet complete and
will be expanded as the core APIs are ported and it is decided
exactly what will be considered to be the core API.

Partially implements blueprint v3-api-extension-framework

Change-Id: Ic2c1d5cd6836ee18819106558880682cf4cda487
Handle security group quota exceeded gracefully

Handle the security group quota exceeded more gracefully so a
traceback is not generated in the log files when this error
occurs. This does not change the API as the HTTP status code
returned remains the same.

Fixes bug 1182859

Change-Id: Id3ac513048751759f01b328482b4f02e26ff66ee