Add log_handler to implement the publish_errors config option

Synced from Olso.
Commit Id is Ibf28ba17f81b5ddeda653f6a3501ecac18ffcd42

This handler sends all error logs as rpc notifications.
Note: when using publish_errors you also need to set the
notification_driver option too.

Change-Id: Ic97b9c8b84d51c84a53ace5b86cc2e72c744f136
Closes-bug: #1240349
(cherry picked from commit 1c4e279e187d1b864680c9be7849130cb7f12801)
Fix migration 185 to work with old fkey names

Due to different versions of MySQL it is possible to end up with
different FK's then what is expected. For example the FK constraint
on instance_info_caches changed from instance_info_caches_ibfk_1
to instance_info_caches_instance_uuid_fkey and on virtual_interfaces
from virtual_interfaces_ibfk_1 to
virtual_interfaces_instance_uuid_fkey meaning databases who have
upgraded from before folsom may have the old fkey.

This patch adds a check for those with the old fkey name to make sure
it is dropped before changing unique constraints (as is required by
MySQL). It also fixes the fkey name to be as defined in 133 if it
wasn't before (which will persist on downgrade being consistent with
those who have come in after folsom).

Closes-Bug: #1245502

Change-Id: Ib0dcd04fcb9ed776c76d40561181abad2e14f76f
(cherry picked from commit c620cafb700ca195db0bd0ef9d62a0c9459bdc38)
Add context as parameter for resume

Now for KVM when resume an instance with block storage, nova
compute will throw exception and failed to resume the VM.

The root cause is that when resume a VM with block storage,
libvirt driver needs to call conductor via rpcapi to update
block device, but the function of resume() do not have context,
this will cause RPC api failed.

Change-Id: I712777ed1d893a2b6463d30c407b0a677e37b602
Closes-Bug: #1241337
(cherry picked from commit de4158861000dedc204314c545cc5682aa38f5f1)
Allow compute manager to accept 3.0 messages

We are currently trying to support live upgrades from havana to
icehouse.  This change is required to make that happen.  In a live
upgrade situation, you have icehouse nova infrastructure, with a mix of
havana and icehouse compute nodes as you roll through and upgrade them.
While in this situation, the havana compute nodes must be able to accept
3.0 messages.

Accepting 3.0 messages is trivial to do and extremely low risk.  It does
not change the behavior of a havana-only deployment at all.  The only
additions is some pass-through code to the current compute manager.
This code will only get hit during a live upgrade situation.

Note that there is one other change needed to complete this.  With only
this patch, live upgrades are supported, but any operations that require
compute nodes to talk to each other will not work.  This includes
resizes and migrates.  That would be unfortunate, since live migrations
are something that people *definitely* want to be able to do during an
upgrade procedure.

The additional changes needed to allow resizes and migrates during live
upgrades are included in another patch.

Partial-bug: #1258253
Change-Id: Ia801609b6b3fe88e0d0b16f4c0d37bf25dbfa842
Allow resizes and migrates during live upgrade

This patch completes the changes needed to havana's compute rpc api to
allow live upgrades to icehouse.  Specifically, these changes are needed
to allow resizes and migrates to work.  Before havana compute nodes get
put in a mixed environment with icehouse compute nodes, they need to be
speaking version 3.0 messages.

Change-Id: If09bd38c9d8c3beb5b95107c497699dec47aa3ef
Closes-bug: #1258253
Fix performance of Server List with Neutron for Admins

https://review.openstack.org/#/c/30048/ optimized the
SecurityGroupsOutputController for the case where the server
list only contains one server, but in all other cases the current
code calls the Neutron driver in a way that makes it retrieve
all ports and security groups visible to the tenant.

For users with a Neutron admin role this retrieves all ports
and all SecGroups in the system, which on a large system is a
major performance issue and often leads to client timeouts.
Normally these users have further qualified their query to a
specific tenant or host, or are maybe just trying to get their
own list of servers.

If instead we pass the pre-filtered list of servers to Neutron
it can query for ports associated with just the device IDs of
the servers.  We then use that list of ports to build a
specific list of security groups IDs to query for.

This approach means that we can remove the initial
optimization for a single server case.

Closes-Bug: 1228384

(cherry picked from commit 18c3ac4a8935b9997dbb55181d5dbb5232ef2c27)

Conflicts:
	nova/tests/api/openstack/compute/contrib/test_neutron_security_groups.py

Change-Id: I6f72533056e4f336f7578cc883fd1a125c2048a9
Enforce permissions in snapshots temporary dir

Live snapshots creates a temporary directory where libvirt driver
creates a new image from the instance's disk using blockRebase.
Currently this directory is created with 777 permissions making this
directory accessible by all the users in the system.

This patch changes the tempdir permissions so they have the o+x
flag set, which is what libvirt needs to be able to write in it and

Closes-Bug: #1227027
Change-Id: I767ff5247b4452821727e92b668276004fc0f84d
(cherry picked from commit 8a34fc3d48c467aa196f65eed444ccdc7c02f19f)
Merge "Allow compute manager to accept 3.0 messages" into stable/havana
Merge "Allow resizes and migrates during live upgrade" into stable/havana
Merge "Add context as parameter for resume" into stable/havana
Fix a lazy-load exception in security_group_update()

When security_group_api is nova, the update_security_group api raises http
500 error: group rules fails with lazy load exception. Because the rule
infos of the security group will be return in response body, but the rule
attribute hasn't been preloaded.

Closes-Bug: #1244018
(cherry picked from commit b9b55935cb61cc1726525b1c4963e352a3a81ea9)

Conflicts:
	nova/tests/api/openstack/compute/contrib/test_security_groups.py
Change-Id: Ie07879ac22e4add90f75968778bf334915e8349c
Merge "Fix migration 185 to work with old fkey names" into stable/havana
libvirt: Fix infinite loop waiting for block job

Libvirt may return an empty dict when a block job has completed.
In this case, _wait_for_block_job will return True, causing
its callers to loop.

Affects volume_snapshot_delete, swap_volume, and live_snapshot.

(See http://libvirt.org/git/?p=libvirt.git;a=commit;h=0f9e67bf )

Closes-Bug: #1260123
Change-Id: Iab81299f0ce32a14e46aee0fd389ed29f7e503b8
(cherry picked from commit e3ac20fca7494122a4c0ed2edc00377aa92cb49d)
VMware: unnecessary session reconnection

An error FAULT_NOT_AUTHENTICATED could imply that there is a problem
with the backend or that no values were found (for example there are
no running VMS). In the latter case the connection should not
be restarted.

The patch also ensures that the session is restarted if there are
connection issues.

This issue occasionally causes CI issues with minesweeper.

Closes-bug: #1265512

Co-authored-by: Shawn Harsock <hartsocks@vmware.com>

Conflicts:

	nova/tests/virt/vmwareapi/test_vmwareapi.py

Change-Id: Ibfa83d0b5b93680890aad8c88c041fa344765448
(cherry picked from commit 55296828a7d7a03736a21d846da15571ca22ec9e)
Merge "Enforce permissions in snapshots temporary dir" into stable/havana
Make test_poll_volume_usage_with_data more reliable

This test includes 2 calls to time.time().  It assumes that the result
from the second call may be greater than the first.  This isn't
definietly true and we see periodic failures of this test in the gate as
a result.

Change the test to not worry about the specific time value that gets
set.  We set _last_vol_usage_poll to 0 before calling the function under
test.  We really only need to check that it got set to some value after
calling the function under test.  This tweak should make the test pass
consistently.

Change-Id: I6c01d2958cf671c29622562689a15ca1ddea0a3c
Closes-bug: #1270693
(cherry picked from commit 7e18d67cc9c96a14e0ca24a41820441e41457738)
Re-write sqlite BigInteger mapping test

This test makes sure that our hook that compiles BigInteger() into
INTEGER for sqlite instead of BIGINT is working as we expect.
Unfortunately, we've seen some problems running the sqlite command to
check the schema.

This patch rewrites the test to only use sqlalchemy APIs and avoid
calling out to the sqlite command for verification.

Change-Id: Id3964d0dd5b0ddf08ab745d6878840ff06b2f0dd
Closes-bug: #1271331
(cherry picked from commit 74ade48634e2b72ec0b09792e97ac77dfb5fd999)
Make test_different_fname_concurrency less racy

This test case makes a bad assumption about when a greenthread will run
and signal its done event.  The test code yields once using a sleep and
expects that the done event will get signaled before the test code runs
again.  This may not be the case.  Instead, yield up to 10 times waiting
for the done event to occur.

I was not able to reproduce this failure locally, but we're seeing it
regularly in CI.  It seems that this should resolve it.

Change-Id: I978ae9eef5bb1ac296055dc1613406e4284a625f
Closes-bug: #1270654
(cherry picked from commit a35776d337493b14d2937f0f4a03aa4b119d018b)
Merge "Make test_poll_volume_usage_with_data more reliable" into stable/havana
Merge "Re-write sqlite BigInteger mapping test" into stable/havana