Libvirt: Always teardown lxc container on destroy

This fixes a bug where shutting down a libvirt-lxc based instance
would leak its underlying nbd device. This was happening because
_teardown_container would only get called if the domain was
present. After this fix, _teardown_container will always get called
on a destroy, which ensures the nbd device gets disconnected.

Change-Id: I8d0044fd038fef80568c6c86e34719ef679aa890
Closes-Bug: #1367363
libvirt: abort init_host method on libvirt that is too old

Nova has a declared minimum supported libvirt and the init_host
method will log an error message if it sees an older version is
installed. Unfortunately people are frequently not seeing this
error message and then filing bugs about things that are broken
on older libvirt versions.

Turn the log message into an exception so that nova-compute
process startup is completely aborted so people will see the
problem immediately rather than some unpredictable time later.

Closes-bug: #1376651
Change-Id: I87d5a41145b05cc2be3559ab528497fe8355ba85
Use reasonable timeout for rpc service_update()

nova.servicegroup.drivers.db.DbDriver._report_state() is called every
service.report_interval seconds from a timer in order to periodically
report the service state. It calls self.conductor_api.service_update().

If this ends up calling nova.conductor.rpcapi.ConductorAPI.service_update(),
it will do an RPC call() to nova-conductor.

If anything happens which causes the RPC reply to be lost or
never sent in the first place, by default the RPC code will
wait 60 seconds for a response (blocking the timer-based
calling of _report_state() in the meantime).
This is long enough to cause the status in the database to get old
enough that other services consider this service to be "down".

if rpc_reponse_timeout is smaller than report_interval then we could
use the existing RPC timeout, but wait longer won't hurt. So the patch
didn't check it and only use report_interval.

Change-Id: I88743183bce1a534812cfe6110c3fc2892058c53
Closes-Bug: #1368989
Add more input validation of bdm param in server creation

Adds a check to ensure that block_device_mapping is a list and
return 400 Bad Request rather than 500 as currently happens.

Change-Id: I30551bcbb7d658734da158dbb8dd7d825c526896
Closes-Bug: 1379973
VMWare: Remove unused exceptions

Nova exceptions UseLinkedCloneConfigurationFault and
VMwareDriverConfigurationException now is present in
oslo.vmware library, so remove it and replace all of
its usages by usages of exceptions from oslo.vmware

Change-Id: I692030cff49965fac2d8da1bb76580fe9a607b61
Use response.text for returning unicode EC2 metadata

When serving meta-data that contains unicode, use the webob Response.text
instead of Response.body.  This also adds a test that triggers the bug in
addition to testing that all meta-data offered by the server can be served.

Closes-bug: #1380792

Change-Id: If2ddaa97bbc89cf574205c0327e3caa02dd503fc
Porting SecurityGroup related controller into v2.1

This patch port SecurityGroup, ServerSecurityGroup and SecurityGroupAction
controller into v2.1 security_groups extension.

This patch just move the v2 code into v2.1 and share the unittest. The other
v2.1 related fix and improvement will be addressed by subsequent patchset.

Partially implements blueprint v2-on-v3-api

Change-Id: I93951ce677b74ebed7db1b2fff1f788344806dba
libvirt: remove pointless HostState class

Now that the get_host_stats method is removed, the only user
of the HostState class passes 'refresh=True' every time. Given
this, there is no point in caching the data in the HostState
class. Instead just populate the data directly in the
get_available_resources method.

Blueprint: virt-driver-get-available-resources-object
Change-Id: I19c391a1bae27c3e8d891ecf77ceeaf170299be3
libvirt: fully parse PCI addresses to integer data type

When parsing node device XML, the PCI addresses were left in their
string form, which is unhelpful for later code wishing to use their
raw values. Fully parse them to the integer form instead.

Blueprint: virt-driver-get-available-resources-object
Change-Id: I46b9c33e08122eb7c28601752ecfc652e09175c3
Clean up the naming of PCI python modules

Something that has been bothering me when looking at the PCI code is
that the modules in /nova/pci/ unnecessarily were prefixed with pci_.
For instance, nova/pci/pci_device.py, whereas all other module directory
organizations for all other code in Nova uses the directory name to
demarcate a submodule.

This patch simply gets the nova/pci/ modules in-line with the standard
set everywhere else in Nova and cuts off the pci_ prefix from the module
names.

Change-Id: Ia5ca13f674f15b41e22b8488efe44eb926e5e4eb
Compute: use instance object for vm_state

Use object format instead of dictionary.

TrivialFix

Change-Id: I6573d2d0d95cbce12733ecb9c9f6d829269d2d10
Pull extra_specs_ops tests from test_host_filters

First in a series of patches to uncruft the 2000+ line
test_host_filters.py unit test file. This patch moves the
extra_specs_ops tests into its own test file.

Change-Id: I200265be6fbc2df1877f01531f443cff8337cebf
Separate and refactor affinity filter tests

The unit tests for the scheduler affinity tests were actually *not*
NoDBTestCase tests. In fact, they were relying on the database to create
instance objects instead of just mocking out the call to
nova.compute.api.get_all(). This patch corrects that and separates out
the affinity filter unit tests out of test_host_filters.py.

Change-Id: Ib7268a49e115fbeff7f18c569cecd82b3d3e8cb3
Remove duplicate test

Remove a test case that is a duplicate of what is done by
test_group_details_in_filter_properties.

Change-Id: I6a6c0b3f9474ca82be05e7ff5d47908a9093b119
Related-bug: #1379451
Port update method of cloudpipe_update to v2.1(v3)

This patch adds couldpipe_update's `update` method to cloudpipe controller
add validation schema to cloudpipe, have v2 unit test cases shared between
v2.1 and v2.

Partially implements blueprint v2-on-v3-api

Change-Id: Ibf39d7fa7301bcba6e259ae88a73c20f64b7fc36
Remove unuseful parameter in cloudpipe api(v2/v2.1)

project_id in _get_cloudpipe_for_project is useless, remove it.

Change-Id: Ie8cb7a6d8ce0678f04200c9fa9de58b4c92bd40d
VMware: Remove redundant fake.reset() in test_vm_util

We fake.reset() in setUp(), so no need to do it in tearDown() too.

TrivialFix

Change-Id: Ie6e2afd351eefe2ff15d7ec74b9bceb65a00422d
libvirt: treat suspend instance as an object

The suspend method receives an object in the compute manager. This
method now uses an object in the driver.

Change-Id: Iccbb70553259aa914303d4d48583644b50a2f142
libvirt: use instance object for resume_state_on_host_boot

The compute manager uses a instance object here. Add support to the
driver.

Change-Id: I6f81a62bcc95f7616414a102f1c470bb621b7bca
libvirt: use instance object for attach_volume

The compute manager uses a instance object here. Add support to the
driver.

Change-Id: I32bc269ce392f1e41f0ddb88b912a5a4bfb7bad8