Fix security group race condition while creating rule

Previously, it was possible for someone to create a security group rule
where the rule references another security group. To do this nova would
first create the security group rule and then look up the referenced group
(group_id's) name in order to return that via the api. During this time
it's possible for someone to delete this security group rule and the
referenced group before the call returned resulting in a 404 error being
raised. This patch addresses this issue by looking up the group name
first and then creating the security group rule in order to avoid this
from occuring.

Note: this patch also adds a test to cover the case where an invalid group
id was passed though this does not really add any real additional coverage
to the change largely because the code is currently using global stubs. That
said, the previous coverage should hopefully be sufficient

Change-Id: If58ffa5629ba5166f260379ac47922974de31be0
Related-bug: 1262566
Reduce unit test times for glance

The class TestGlanceApiServers did not inherit
test.NoDBTestCase.

Change-Id: Ieafe300d793685f5b146999383695733eba3da19
Baremetal driver: remove unused states

NULL and INIT were not used since commit c20110d15be37948ddd9ef5f38001328aabf5b1d

Change-Id: I25c34872b100807d691e7e15f7b70be7ad1912b9
VMware: cleanup the constructors of the compute drivers

The 'read_only' parameter was never used by the compute drivers.
The 'scheme' parameter should be passed to the base constructor.

Change-Id: I3abd40a50896664ee274f057ded8ec2460679c82
Add debug log for core_filter

Sometimes operator need information why the host doesn't pass
the check of scheduler, this patch adds information for core
filter if virtual cpu is not enough on the host.

Change-Id: Ie05d9184435a4fc4cff1af26812c01a3cc226fff
Partial-Bug: #1301830
Add support for user_id based authentication with Neutron

Keystone v3 supports non-unique project/user names, so until
the Neutron client provides support for the V3 API and domain
based authentication Nova should allow user ID for admin
authentication.

Change-Id: Idad27ff4206d6ba0bb9aedbbff5172c3144ad3c6
DocImpact: Adds new flag, neutron.admin_user_id
object-ify API v2 availability_zone extension

This change converts the availability_zone extension to use the
Service object instead of direct database access.

Related to blueprint compute-manager-objects-juno

Change-Id: If5baafd8ec85be1ea1cd6580ae10ef3d51366c9c
Remove pause/unpause NotImplementedError API layer

There are 2 kinds of RPC call from API layer to compute layer,
one is cast and another is call. For cast, the RPC message will
be posted and the API service will not wait for the message to
be processed. So it won't be able to catch the exception raised
in compute layer, so catch and handle the exception is useless
and error leading. This patch removes code in API layer for
pause and unpause functions.

Change-Id: I7cbb8179b3eec8496b90ff5cf07397522d52c5de
Fix error status code for multinic

When passing bad body in a request, most APIs return BadRequest response.
However, multinic API doesn't do it.
This patch fixes the error status code and adds a unit test related to
this change.

This patch's idea most came from
following patch: https://review.openstack.org/#/c/107266/

Change-Id: If1ef2077a0e9823dfc04e4ef5a6dfb02daf1a8dc
Add instanceset info to StopInstance response

Currently stopinstance response missing the instanceset
information with InstanceID, current state and previous
state details. It just returns the "True".

As per the AWS EC2 API reference document, the StopInstance
response elements should include the instanceset information
as below:
<StopInstancesResponse xmlns="http://ec2.amazonaws.com/doc/2014-02-01/">
  <requestId>req-a7326465-5ce2-4ed6-ab89-394b38cca85f</requestId>
  <instancesSet>
    <item>
      <instanceId>i-00000001</instanceId>
      <currentState>
        <code>80</code>
        <name>stopped</name>
      </currentState>
      <previousState>
        <code>16</code>
        <name>running</name>
      </previousState>
    </item>
  </instancesSet>
</StopInstancesResponse>

Included the instanceset into stopinstance response elements
and updated the test cases for stopinstance response elements
in nova/tests/api/ec2/test_cloud.py file.

Closes-bug: #1321239

Change-Id: I4d8a6faf2689a7df71920183682fd1e403ce2a42
Restore ability to delete aggregate metadata

Commit I2c778f2237ba5bd2aa8335a0eae80f3aad3e9157 tightened the
input validation for setting the metadata for an
aggregate. However it also accidentally excluded sending
null as the metadata value which has previously allowed the
deletion of the metadata key rather than an update. This
changeset relaxes the input validation to allow deletion of
aggregate metadata again

Change-Id: I54b91c2d421243e7e7521f2352598f73a258b88a
Closes-Bug: 1334846
Send compute.instance.create.end after launched_at is set

Downstream consumers of notifications want to know when an instance was
launched from the compute.instance.create.end notification.  This value
was previously set in the notification but commit
70196c4854e6cc1cb566dc9fc5b1a3397a4b69ab changed things so that it was
no longer set.  It will now be set, and is tested to be set.

Change-Id: If5ac00529d52e3898cda7fd942535ed3ba2039d5
Closes-bug: 1338736
Don't remove delete_on_terminate volumes on a reschedule

When cleaning up volumes before a reschedule if delete_on_terminate is
True the volume would be deleted.  That's not the desired behavior so
the volume cleanup has been moved to take place when a build is aborted.

Change-Id: I142370c0555495b4d51736f4f6b8070a3c112a59
Closes-bug: #1336127
Add decorator expected_errors to V3 servers core

All v3 api should use expected_errors to prevent unexpected exception
raised. This patch add decorator expected_errors for v3 servers core.

Change-Id: Ia1fa09d6cf68a65c85ac5fd2aff9b9202d76e7ca
Closes-Bug: #1343778
Add decorator expected_errors for v3 attach_interfaces

All v3 api should use expected_errors to prevent unexpected exception
raised. This patch add decorator expected_errors for v3 attach_interfaces.

This patch also remove update action. Let the wsgi return HTTPNotFound for
it.

Change-Id: Icefe9f9adfa7ad01bc3ab11dcb7c132702390d04
Remove unnecessary error log in cell API

There are unnnecessary error logs in cell API. It tried to log
error for invalid input which is not necessary because we already
raises exception for the bad request.
This patch removes them in both v2/v3 API.

Change-Id: Idd239903aa830f5d8fe50336dfa63a43b617584f
Commit quota when deallocate floating ip

It seems that when you allocate a floating-ip in a tenant with nova-network,
its quota is never returned after calling 'nova floating-ip-delete'
even though 'nova floating-ip-list' shows it gone.
This behavior applies to each tenant individually. The gate tests are passing
because they all run with tenant isolation.

The root cause of the problem is cooperation between
commit 23a27e47 and cbbb9de5. db layer code return floatingip_ref
but object layer didn't. This patch fixed the problem by adding return
value from object layer.

Change-Id: Ide1a338b6c33676311028e8738150e146324a8ee
Closes-Bug: #1347156
Keep resizing&resized instances when compute init

During compute manager startup init_host is called. One of the
functions there is to delete instance data that doesn't belong
to this host i.e. _destroy_evacuated_instances.
But this function only checks if the local instance belongs to
the host or not. It doesn't check the task_state or vm_state.

In Resize function, user may want to revert or confirm the resize
operations so the instance on source and dest compute node should
be kept. so for RESIZE_MIGRATING, RESIZE_MIGRATED task states and
RESIZED vm state instances, they should be kept in compute node
when the compute restart. This patch adds check for the task
state and vm state before delete the instances.

Closes-Bug: #1330503

Change-Id: I723fa4a8823019391ea83aa189096531032adab1
Add debug log for pci passthrough filter

Sometimes operator need information why the host doesn't pass
the check of scheduler, this patch adds information for pci
passthrough filter if it falied to pass check.

Change-Id: Ic8fe13174fda5c75ce95fc0b64391ae081b36c6d
Partial-Bug: #1301830
Downgrade the warn log in network to debug

Downgrade the log from warn to debug for overquota.
Didn't remove it because it's called internally and it might
be helpful for operater to take a look.

Change-Id: I1d3c27a8d97085d9919180a28018b234392f28e4