Make nbd reservation thread-safe

Avoid the situation where two local threads choose the same nbd number
for injecting files into the instance. If this happened quickly enough
nova was left with one qemu-nbd hanging and blocking the device forever.

Fixes bug 1207422

Change-Id: I18864b19ebd30669534e45ca7a50e12f61207302
Don't need to init testr in run_tests.sh

In run_tests.sh, function init_testr will initialize testr if the
directory .testrepository is not existed. Actually, testr will do
the check before run the test:

In Python package testrepository, setuptools_command.py:Testr.run

68 def run(self):
69     """Set up testr repo, then run testr"""
70     if not os.path.isdir(".testrepository"):
71         self._run_testr("init")

So, init_testr can be removed safely.

Fixes Bug #1211098

Change-Id: I4bd2ef3dc107281892732efced268eb3b7c90cbb
Fix HTTP response for PortInUse during boot (v3 API)

If an in use port is passed as a network parameter in the request, Nova API
should raise an HTTPConflict (409) error. This was fixed in commit
48b31c9a1f5d2a45d881e70c6223411c5abbcb72 for the V2 API but not the V3 API.

Fixes bug 1212098

Change-Id: I94429f2c04cf81cb88a8739f65a49aa7ca0d49bd
Raises masked AssertionError in _test_network_api

The _test_network_api method was catching an AssertionError to do some
more checking but failed to raise the original error if the checking
didn't hit any matches. This patch fixes the test to raise the original
assertion error.

Closes-Bug: #1212017

Change-Id: I33147d18e33cd4babfc3f5cb572007bc7e4c07e6
Implement front end rate-limiting for Cinder volume

Cinder adds Quality-of-Service control interface to allow admin to
specify QoS parameters as well as the place where these parameters
should be consumed (Front-end: Nova Compute, Back-end: Cinder
Volume, or Both) for volume types.  KVM and other hypervisor is
known to be able to do disk rate-limit control (one form of QoS)
for virtual disks, this patch enables Nova Compute to extract
rate-limit parameters for volumes while attaching, currently
libvirt only.

implement bp volume-rate-limiting

Change-Id: Ibcc8f15ab744600e6f814b46935f8022d78074fc
Fix untranslated log messages in libvirt driver

Fixes bug 1213040

Change-Id: Ib3b7a46c312b88778cc6e872a2c19061a5774125
fix volume swap exception cases

Currently when the target instance id doesn't match with the instance_id of the
volume to be replaced VolumeUnattached is raised, and InvalidVolume would be a
fittest response.
In the other hand, currently, when the volme to be replaced is not attached at
all InvalidVolume is raised, and VolumeUnnatached would be a fittest response.

Change-Id: I661b97762af70765ab8111313d24cf1a175f7e6c
Added test case in test_migrations for migration 208

This patch is to add test case in test_migrations.py for migration 208.
Migration 208 was brought by commit 74b024f75403c168b5c69ee67a16d1053965e5db
(Change-Id: I72371260159c9fdffcdb57209095fc103b6dd79e)

Change-Id: I153abf260716c8af61d7ccf554d91ba80f559e60
Signed-off-by: Shane Wang <shane.wang@intel.com>
Make xenapi capabilities['enabled'] use service enabled

filters.compute_filter is enabled by default along with libvirt, but the
capabilities['enabled'] section of it is only supported by XenAPI.
This used to work by adding a parameter to capabilities, but since
the capabilities RPC is being removed, use conductor.service_update to tell
the scheduler which host is disabled.

service_update is already the way to tell the scheduler about other
disabled hosts, so to keep backwards compatibility keep host_enabled but
use service_update underneath.

Part of bp no-compute-fanout-to-scheduler

Change-Id: Ica1919cfe9a2402b2107c557c10d47e01a6c430b
Correct the status code to 201 for create v3

Partially implements bp nova-v3-api

DocImpact

Change-Id: Ie7413f0f6473ad12b84f84688c7c8c377ebf39e1
Add RBD supporting to libvirt for creating local volume

The block storage in OpenStack is divided into Nova and Cinder, Nova is
responsible for local volume such as root, swap and ephemeral partition. Cinder
is responsible for normal volume(global volume?). This mode will make difficult
for planning storage allocation. In order to improve the availability of
storage capacity and reduce the complexity of storage allocation, implementing
the unified storage for local volume in Nova and volumes in Cinder is
necessary.

Now Nova supports lvm backend for creating local volume. Ceph as a Cinder
backend we want to implement it too in Nova.

This patch add RBD backend to libvirt image backend, and users can use specify
rbd as the source of root, ephemeral and swap partition.

Implement: blueprint bring-rbd-support-libvirt-images-type
Change-Id: Id97de5c9af4ddc5c58493a5cef21b52386afac3e
Object support for instance groups

The patch set adds object support to the instance groups.
This is required for: https://review.openstack.org/#/c/30028/

This is part of the blueprint instance-group-api-extension.

Change-Id: I599016b7a15d6045ae5129352eacc60a37ac7106
Fix aggregate creation/update with null or too long name

Fixes aggregate creation so an attempt to create an aggregate with
a null name results in a 400 Bad Request rather than 409 Conflict.
This was fixed for the V2 api in 43f9028f84ffec9b465bec10025ba75266c1bc22
and is fixed in this changeset for the V3 api

For both the V2 and V3 APIs adds a check to ensure that the aggregate
name is not longer than 255 characters.

Also fixes null or too long names when updating an aggregate

Fixes bug 1212240

Change-Id: I644a8a46f0b75e3717ecafec1889ce6316ca18c6
Link Service.compute_node with ComputeNode object

This is the third in a three-step dance to implement a Service
object, a ComputeNode object, and finish the relationship with
the circular reference from Service to ComputeNode. It adds
Service.compute_node, a nested object relationship that is either
lazy-loaded on access, or pre-loaded if a database query
includes it. This should exactly match the behavior of the
SQLAlchemy objects that we have now.

Note that this requires a new DB API method to look up a
ComputeNode object from its service_id. This was previously
provided by the SQLAlchemy relationship between the two objects.

Related to blueprint compute-api-objects

Change-Id: If62d443e34a3c8ce9bc83008fc69821494686bc7
Revert task state when terminate_instance fails

Calls to terminate_instance leave the instance task state as-is when
they raise an exception - this was introduced in Ie5701e5c as a
work-around for a bug with quota accounting, but introduced bugs with
power-state syncing. Ib72de1a4 removed the need for the original
work-around, and this change reverts Ie5701e5c.

Ie5701e5c worked around a bug in quota accounting by using the instance's
task state as a signal to communicate between the nova.compute.manager
and nova.compute.api modules: if the manager's terminate_instance method
failed, it would leave the instance task state as DELETING. If the user
retried the delete, the api would check the task state: if the
instance's task state was already DELETING, it would not decrement the
user's quota a second time.

Unfortunately, leaving an instance with an active task had the
side-effect of preventing the VM power-state from synchronizing, wedging
the instance (bug 1177584 and bug 1212420).

Ib72de1a4 changed the quota accounting mechanism, moving the quota
handling code out of the api module. The api code for the work-around to
check the DELETING task state was removed. However, the code for the
work-around in the manager was never cleaned up.

This change cleans up the nova.compute.manager work-around code, making
terminate_instance revert the task state to None upon failure.
Power-state syncing should work again, and quota handling is unaffected
by the change.

Fixes: bug 1177584, bug 1212420
Change-Id: I42c68d16dca56703d6b192bb452883e4d5ca80df
Removes fixed ips extension from V3 API

The fixed ip extension was one of the first ported to the V3 API
but a later decision around not supporting nova network in the V3 API
means that this extension is not required for V3. The fixed ip extension
only supports nova network, not neutron and the same functionality
can be accessed directly through neutron.

Also changes some extension tests to use a different extension
for white/black list testing

Partially implements bp nova-v3-api

Change-Id: Ie589170e863ee489661e789c5134ed8349ced279
Fix typo in baremetal docs

instanse_type_extra_specs=>instance_type_extra_specs

Change-Id: I8075d1e535e74616d7f1c05ba69f1c9f4c9d4293
Add volume snapshot APIs to driver interface

Add volume_snapshot_create and volume_snapshot_delete to the virt driver
interface.  This is needed to be able to do hypervisor assisted snapshot
operations.

Part of blueprint qemu-assisted-snapshots

Change-Id: I933e000ee56430365f9e380196714f43a85f147f
Add db.block_device_mapping_get_by_id

Add a new db API method for getting a block device mapping for a given
volume ID.  This is needed when Cinder calls over to Nova asking nova to
complete a snapshot operation on its behalf.

Related to blueprint qemu-assisted-snapshots

Change-Id: If3f011e2db14235f4137c9d9cb610f0c4cd7954a
Update availability_zone on time if it was changed

The problem is that when the availability_zones for one host
aggregation was updated, the VM attribute of availability_zones
which belong to the host in the availability_zones was
not updated on time.

If updated values include availability_zones, then the cache
which stored availability_zones and host need to be reset.

Fix bug 1203250

Change-Id: Ib3f08fbe3c74d5aa101da78faed43b0bc03cb988