Fix security group race condition while creating rule

Previously, it was possible for someone to create a security group rule
where the rule references another security group. To do this nova would
first create the security group rule and then look up the referenced group
(group_id's) name in order to return that via the api. During this time
it's possible for someone to delete this security group rule and the
referenced group before the call returned resulting in a 404 error being
raised. This patch addresses this issue by looking up the group name
first and then creating the security group rule in order to avoid this
from occuring.

Note: this patch also adds a test to cover the case where an invalid group
id was passed though this does not really add any real additional coverage
to the change largely because the code is currently using global stubs. That
said, the previous coverage should hopefully be sufficient

Change-Id: If58ffa5629ba5166f260379ac47922974de31be0
Related-bug: 1262566
VMware: cleanup the constructors of the compute drivers

The 'read_only' parameter was never used by the compute drivers.
The 'scheme' parameter should be passed to the base constructor.

Change-Id: I3abd40a50896664ee274f057ded8ec2460679c82
Add debug log for core_filter

Sometimes operator need information why the host doesn't pass
the check of scheduler, this patch adds information for core
filter if virtual cpu is not enough on the host.

Change-Id: Ie05d9184435a4fc4cff1af26812c01a3cc226fff
Partial-Bug: #1301830
Fix error status code for multinic

When passing bad body in a request, most APIs return BadRequest response.
However, multinic API doesn't do it.
This patch fixes the error status code and adds a unit test related to
this change.

This patch's idea most came from
following patch: https://review.openstack.org/#/c/107266/

Change-Id: If1ef2077a0e9823dfc04e4ef5a6dfb02daf1a8dc
Restore ability to delete aggregate metadata

Commit I2c778f2237ba5bd2aa8335a0eae80f3aad3e9157 tightened the
input validation for setting the metadata for an
aggregate. However it also accidentally excluded sending
null as the metadata value which has previously allowed the
deletion of the metadata key rather than an update. This
changeset relaxes the input validation to allow deletion of
aggregate metadata again

Change-Id: I54b91c2d421243e7e7521f2352598f73a258b88a
Closes-Bug: 1334846
Add decorator expected_errors to V3 servers core

All v3 api should use expected_errors to prevent unexpected exception
raised. This patch add decorator expected_errors for v3 servers core.

Change-Id: Ia1fa09d6cf68a65c85ac5fd2aff9b9202d76e7ca
Closes-Bug: #1343778
Add decorator expected_errors for v3 attach_interfaces

All v3 api should use expected_errors to prevent unexpected exception
raised. This patch add decorator expected_errors for v3 attach_interfaces.

This patch also remove update action. Let the wsgi return HTTPNotFound for
it.

Change-Id: Icefe9f9adfa7ad01bc3ab11dcb7c132702390d04
Remove unnecessary error log in cell API

There are unnnecessary error logs in cell API. It tried to log
error for invalid input which is not necessary because we already
raises exception for the bad request.
This patch removes them in both v2/v3 API.

Change-Id: Idd239903aa830f5d8fe50336dfa63a43b617584f
Keep resizing&resized instances when compute init

During compute manager startup init_host is called. One of the
functions there is to delete instance data that doesn't belong
to this host i.e. _destroy_evacuated_instances.
But this function only checks if the local instance belongs to
the host or not. It doesn't check the task_state or vm_state.

In Resize function, user may want to revert or confirm the resize
operations so the instance on source and dest compute node should
be kept. so for RESIZE_MIGRATING, RESIZE_MIGRATED task states and
RESIZED vm state instances, they should be kept in compute node
when the compute restart. This patch adds check for the task
state and vm state before delete the instances.

Closes-Bug: #1330503

Change-Id: I723fa4a8823019391ea83aa189096531032adab1
Inject expected results for IBM Power when testing bus devices

Same as in https://review.openstack.org/#/c/93621 this patch
inject expected values when the test is running on IBM Power
platform, since the ide bus isn't supported on IBM Power,
and libvirt translate all ide calls directly to scsi
when it's running in power platform

Change-Id: Ibebc96e71a8ba21ae8634e9e22de1cd28f3a7990
Catch CannotResizeDisk exception when resize to zero disk

When old flavor's root_gb is not equal 0 and new flavor's root_gb is 0,
the resize() in nova.compute.api will raise CannotResizeDisk.

Move up the new_instance_type check before using.

Closes-Bug: #1340159

Change-Id: I4abf93530cf919af50a88d6049019fb745547257
libvirt: add support for memory tuning in config

Add a libvirt config class for dealing with the XML elements
under the <memtune> element of the guest schema

Blueprint: virt-driver-large-pages
Change-Id: I141fcf4fd95f61f7fc3a5d0209dc2b1319ede941
Add ibmveth model as a supported network driver for KVM

ibmveth (spapr-vlan) is supported by KVM/Qemu on ppc64. This change
allows users to change hw_vif_model='spapr-vlan' property to use
ibmveth driver.

Change-Id: Ibec6abad458c5cb15df57afa05f7f0742e4feb14
VMware: improve log message for attachment of CDROM

Commit 3c59d998c5e6b80549053b01f919ac318cd31b13 config drive
support. The log message makes use of the instance_name which is
not unique. The may be confusing when debugging.

TrivialFix

Change-Id: Iff7f4f938f4980f0c0bf2dc3017ed741453a69f5
Scheduler: throw exception if no configured affinity filter

If the scheduler hint indicates that the scheduling should perform
either anti-affinity or affinity scheduling and the relevant filter
is not configured then a NoValidHost exception will be thrown.

This is valuable if an existing OpenStack installation is running
and these filters are not defined after an upgrade.

Change-Id: I79bb44ad7481b3ff924687a8d6afdd6c715c0b59
Closes-bug: #1302238
libvirt: removing lxc specific disk mapping

Currently, LXC specific disk mapping is being created, preventing
the attached volumes from being added to the LXC disk mapping.
Removing this part, to allow volumes to be included in mapping.

Related-Bug: #1269990
Change-Id: I97bcff558b6d8876a10adb36c19ab3f4c972a708
libvirt: add tests for _live_snapshot and _swap_volume methods

The _live_snapshot and _swap_volume methods do a non-trivial
sequence of libvirt API calls which should be validated in the
unit tests

Closes-bug: #1346327
Change-Id: I14bfa08f0dbb4eaa7c4f069f70c257ba7605989b
Stop depending on sitepackages libvirt-python

We can install libvirt-python bindings from pypi now because the libvirt
devs are supporting pypi packages and the version of libvirt we are
testing nova master against is new enough to link against that pypi
package.

Making this switch will better isolate nova test environments as all
dependencies can be installed without any site packages leaking into
virtualenvs. It will also make the dependency on libvirt-python a bit
more explicit.

Change-Id: I2890702869f05a02ad5e2ecb419db06433231b36
Allow dhcp_server to be set from new field

Now that dhcp_server and gateway can be different, update dnsmasq
to hand out the gateway if we have a single ip address and they
don't match.

Partially-implements blueprint better-support-for-multiple-networks

Change-Id: Id040ce1fb0f889418815568ff6c80402b69bf09e
Only use dhcp if enable_dhcp is set on the network

Also removes an unused call to init_host if there is no network
to setup. The signature of this call was wrong so clearly it was
not being used.

Partially-implements blueprint better-support-for-multiple-networks

Change-Id: I33c86b1500f9a69081b4c6ecbe79151f10f69cea