Fix unset extra_spec for a flavor

A flavor extra_specs key cannot be deleted using 'nova flavor-key'.
Updates to a flavor's extra_specs are not being registered by
NovaObject, such that when NovaObject.obj_what_changed is called,
no changes in the extra_specs field are picked up. In order for
changes in extra_specs to be picked up, the updated extra_specs
has to be compared against the _orig_extra_specs. The same approach
also applies to the flavor's projects, where the updated projects
has to be compared against the _orig_projects.

Also, if you delete the last remaining extra_specs key, nova would
throw an sqlalchemy warning, i.e. 'SAWarning: The IN-predicate on
"instance_type_extra_specs.key" was invoked with an empty sequence'.
This was caused by calling db.flavor_extra_specs_update_or_create
with an empty dict of keys to add or update. This patch fixes it
calling db.flavor_extra_specs_update_or_create only on a non-empty
dict of keys.

Change-Id: Icea492134239296f1570b2a4208eb0868315aa5f
Closes-Bug: #1340885
Remove the AUDIT log message about loaded ext

Finally, I got to a boiling point when having to scroll through endless
and pointless log.AUDIT messages about "Loading extension XXX" in the
startup messages for the os-api endpoints. This patch gets rid of this
extreme annoyance.

Change-Id: Ia47f6afae89d145ade2a940d4720f5977fde2204
Sync network_info if instance not found before _build_resources yield

If instance is deleted before _build_resources yield, there won't
waiting for network_info sync finish.

So there is race case, when the exception catching for
'InstanceNotFound' in do_build_and_run_instance invoke
'_cleanup_allocated_networks', the '_allocate_network_async' didn't
create port yet. After '_cleanup_allocated_networks' finished,
'_allocate_network_async' create new port for the instance, but there
isn't any cleanup for the new port anymore.

Change-Id: I5027a1aaeb9a2106f7982fd0422f88c4caa20ffb
Related-Bug: #1357055
Merge "Be less confusing about notification states"
Reduce the log level for the guestfs being missing.

There are platforms where we can never reasonably expect libguestfs
to be functional (FreeBSD, libvirt / lxc, etc). Reduce the log
level of the message recording that libguestfs is unavailable to
reduce operator confusion.

As discussed on openstack-dev at:

  http://lists.openstack.org/pipermail/openstack-dev/2014-September/046910.html

Change-Id: I4bd7ed4fbffb8ac22aea4285fe74ee0d78e1581c
Add missing instance action record for start of live migration

We should add action record for live migration as we did for resize,
start,etc.

Change-Id: I6838727bf51f4bcdba0010c72bd2d5132bcf2cc3
Merge "Add further debug logging for multiprocess test"
Add test case for vim header check

There was not a unit test case for the vim header hacking
check.  This adds a test case.

Change-Id: I66889819b4d0b62ea54da327a3724c523a0e4fcb
consistently set status as REBUILD when rebuilding

Consider an instance is in SHUTOFF status(stopped). When a rebuild
is issued on this instance, the status does not display as REBUILD.

Similarly, when a rebuild is issued on an instance and the operation
fails, the instance turns to ERROR state. Now if we issue a rebuild
again, the status does not display as REBUILD.

This patch fixes the above mentioned issues by displaying the correct
status i.e REBUILD.

Change-Id: Id11c9d31683a049c61fd751dc0ddabaf22c96cb9
Closes-Bug: #1362733
Fix start/stop return active/stopped immediately in EC2 API

start/stop action is async. But the EC2 API is hard code the status
as start/stop finished immediately.

This patch fix start action return pending status, and stop action
return stopping status.

Change-Id: I0cf6db1dcb17c48a12fd62b0d707afd124b96b26
Closes-Bug: #1373230
Merge "Remove the AUDIT log message about loaded ext"
Merge "Revert "libvirt: Uses correct imagebackend for configdrive""
Return vcpu pin set as set rather than list

The current implementation of the libvirt driver makes mixed assumptions
about the vcpu pin set returned from hardware.get_vcpu_pin_set(). Most
places in the code assume its a set and perform set operations on the
structure. However a few  places assume it's a list. Given the mixed
assumptions about the structure type, the existing code was trying
to perform set operations on a list.

This patch changes the get_vcpu_pin_set() method to return a set
rather than a list and handles any edge cases where consumers need
a list. It also updates any relevant unit tests accordingly.

Change-Id: I66d5cbc0e2d370d9d2d2ab2bad2c5b348bedba6c
Closes-Bug: 1372829
XenAPI: clean up old snapshots before create new

Before taking a new snapshot, we now inspect the VDI chain to check for
old snapshots.

Those old snapshots can be left behind from resize, migrates, and
snapshots that get interrupted if you kill the nova-compute process.

If you leave these snapshots behind, the hypervisor disk will get too
full, as VDI chains grow much bigger than you would expect. In addition
snapshots start to grow in size, causing issues copying the snapshot
into storage.

Closes-Bug: #1371072

Change-Id: Ice3741213438c7b4a7fda1e3e0a6dd7faca0fd04
add time to logging in unit tests

It's quite difficult to debug unit test failures when you don't have
any idea when the log messages happened relative to each other. Update
the logging fixture so that we are actually writing out time into our
log messages.

Change-Id: I704792f10d77345246d528bd9abf2e0c46b6df72
remove test_multiprocess_api

This test suite should be removed for the following reasons.

1) it's not a unit test

2) it's failing randomly, a lot, and depends on timing of the host for
process start and signal handling which are provably not predicable on
slower machines

3) it's triggering testr bugs in the ways that it fails which means we
don't actually get any subunit in the fail scenario, which makes it
actually impossible to move forward on fixes.

Change-Id: Ieb7c02115c12cd9f6c9fa67691a643aed7632784
Closes-Bug: #1357578
Give context to the warning in _sync_power_states

The "Found x in the database and y on the hypervisor"
warning in the logs doesn't give any context to what
is happening when that is logged, this adds the context.

Change-Id: I8800006508b2e18de657aa0a8d135bcd17cb89f1
libvirt: convert CacheConcurrencyTestCase to avoid DB usage

The CacheConcurrencyTestCase test case does not do anything DB
related. Convert it to NoDBTestCase to avoid DB setup penalty.
This improves perf from 1.7s to 0.2s

Partial-bug: #1369516
Change-Id: I8ef124e9f5d86a990722019978c725c955ffb1ea
Reduce the scope of RT work while holding the big lock

This makes the RT.update_available_resource() method do what it
can in the way of prep work before grabbing the lock to update
the database stats.

Change-Id: I1132e2ab99427688529566c1a1fffb33eab07f3f
Related-Bug: #1371677
Neutron: Atomic update of instance info cache

In the Neutron network API implementation, there is a race condition
between a thread performing periodic task to read and heal instance
network info cache and another thread servicing interface-attach or
interface-detach calls. This patch ensures that instance info cache is
read and then updated in a synchronized block to ensure atomicity.

Please see the bug report for more details.

Change-Id: Ifc76f2b1cce834b3c9927359ac9b957bc9f9c65f
Closes-Bug: #1326183
Co-Authored-By: Dan Smith <dansmith@redhat.com>