Add log_handler to implement the publish_errors config option

Synced from Olso.
Commit Id is Ibf28ba17f81b5ddeda653f6a3501ecac18ffcd42

This handler sends all error logs as rpc notifications.
Note: when using publish_errors you also need to set the
notification_driver option too.

Change-Id: Ic97b9c8b84d51c84a53ace5b86cc2e72c744f136
Closes-bug: #1240349
(cherry picked from commit 1c4e279e187d1b864680c9be7849130cb7f12801)
Fix `NoopQuotaDriver.get_(project|user)_quotas` format

The quota API extension expects `get_project_quotas` and `get_user_quotas` to
return a dictionary where the value is another dictionary with a `limit` key.

The `DbQuotaDriver` adhered to this spec, but the `NoopQuotaDriver` didn't.

This fixes the `NoopQuotaDriver` to return the results in the correct format.

Fixes bug 1244842

Change-Id: Iea274dab1c3f10c3cb0a2815f431e15b4d4934b1
(cherry picked from commit 711a12b4029cd1544d26d147d8a67e110e056124)
Fix interface-attach removes existing interfaces from db

The following commit 394c693e359ed4f19cc2f7d975b1f9ee5500b7f6 changed
allocate_port_for_instance() to only return the ports that were created
rather than all of the ports on the instance which broke the attach-interface
code.

This patch fixes this issue by remove the sync decorators from:
allocate_for_instance, allocate_port_for_instance, and
deallocate_port_for_instance to just _build_instance_nw_info which
is called in all these cases instead.

Closes-bug: #1223859

(cherry picked from commit 1957339df302e2da75e0dbe78b5d566194ab2c08)

Conflicts:
	nova/network/neutronv2/api.py

Change-Id: I66eb0c0ab926e0a8d1e2c9cfe1f7fd579ea3aa27
libvirt: Allow delete to complete when a volume disconnect fails

If an instance which was booted from a volume fails to spawn, the
instance is left in an ERROR state.  If the failure is in the networking
stage Nova will not have attached the volume via the Cinder driver,
and so the volume remains in an available state.  In this state the
volume may be deleted or assigned to another instance.
During the subsequent delete an exception may be thrown from
the cinder driver which prevents the instance from completing the
deletion process leaving the instance stuck in Error and unable to
be deleted.

Volume exceptions should be logged and ignored when the instance
is being deleted.

Closes-Bug #1222979

(cherry picked from commit 506a8f58cf4b8cecf90b647c7deba47da2a4dfec)
Conflicts:
	nova/tests/virt/libvirt/test_libvirt.py

Change-Id: Icb3796b0ddba25cf344953a649b2e762fab6d782
Fix performance of Server List with Neutron for Admins

https://review.openstack.org/#/c/30048/ optimized the
SecurityGroupsOutputController for the case where the server
list only contains one server, but in all other cases the current
code calls the Neutron driver in a way that makes it retrieve
all ports and security groups visible to the tenant.

For users with a Neutron admin role this retrieves all ports
and all SecGroups in the system, which on a large system is a
major performance issue and often leads to client timeouts.
Normally these users have further qualified their query to a
specific tenant or host, or are maybe just trying to get their
own list of servers.

If instead we pass the pre-filtered list of servers to Neutron
it can query for ports associated with just the device IDs of
the servers.  We then use that list of ports to build a
specific list of security groups IDs to query for.

This approach means that we can remove the initial
optimization for a single server case.

Closes-Bug: 1228384

(cherry picked from commit 18c3ac4a8935b9997dbb55181d5dbb5232ef2c27)

Conflicts:
	nova/tests/api/openstack/compute/contrib/test_neutron_security_groups.py

Change-Id: I6f72533056e4f336f7578cc883fd1a125c2048a9
Enforce permissions in snapshots temporary dir

Live snapshots creates a temporary directory where libvirt driver
creates a new image from the instance's disk using blockRebase.
Currently this directory is created with 777 permissions making this
directory accessible by all the users in the system.

This patch changes the tempdir permissions so they have the o+x
flag set, which is what libvirt needs to be able to write in it and

Closes-Bug: #1227027
Change-Id: I767ff5247b4452821727e92b668276004fc0f84d
(cherry picked from commit 8a34fc3d48c467aa196f65eed444ccdc7c02f19f)
VMware: fix bug when more than one datacenter exists

In the case that there was more than one datacenter defined on the VC,
then spawning an instance would result in an exception. The reason for this
was that the nova compute would not set the correct datacenter for the
selected datastore.

The fix also takes care of the correct folder selection. This too was a
result of not selecting the correct folder for the data center.

The 'fake' configuration was updated to contain an additional data
center with its on datastore.

Closes-Bug: #1180044
Closes-Bug: #1214850

Co-authored-by: Shawn Harsock <hartsocks@vmware.com>

(cherry picked from commit a25b2ac5f440f7ace4678b21ada6ebf5ce5dff3c)

Conflicts:

	nova/tests/virt/vmwareapi/test_vmwareapi.py
	nova/virt/vmwareapi/fake.py

Change-Id: Ib61811fffcbc80385efc3166c9e366fdaa6432bd
Fix a lazy-load exception in security_group_update()

When security_group_api is nova, the update_security_group api raises http
500 error: group rules fails with lazy load exception. Because the rule
infos of the security group will be return in response body, but the rule
attribute hasn't been preloaded.

Closes-Bug: #1244018
(cherry picked from commit b9b55935cb61cc1726525b1c4963e352a3a81ea9)

Conflicts:
	nova/tests/api/openstack/compute/contrib/test_security_groups.py
Change-Id: Ie07879ac22e4add90f75968778bf334915e8349c
libvirt: Fix infinite loop waiting for block job

Libvirt may return an empty dict when a block job has completed.
In this case, _wait_for_block_job will return True, causing
its callers to loop.

Affects volume_snapshot_delete, swap_volume, and live_snapshot.

(See http://libvirt.org/git/?p=libvirt.git;a=commit;h=0f9e67bf )

Closes-Bug: #1260123
Change-Id: Iab81299f0ce32a14e46aee0fd389ed29f7e503b8
(cherry picked from commit e3ac20fca7494122a4c0ed2edc00377aa92cb49d)
VMware: unnecessary session reconnection

An error FAULT_NOT_AUTHENTICATED could imply that there is a problem
with the backend or that no values were found (for example there are
no running VMS). In the latter case the connection should not
be restarted.

The patch also ensures that the session is restarted if there are
connection issues.

This issue occasionally causes CI issues with minesweeper.

Closes-bug: #1265512

Co-authored-by: Shawn Harsock <hartsocks@vmware.com>

Conflicts:

	nova/tests/virt/vmwareapi/test_vmwareapi.py

Change-Id: Ibfa83d0b5b93680890aad8c88c041fa344765448
(cherry picked from commit 55296828a7d7a03736a21d846da15571ca22ec9e)
Merge "Enforce permissions in snapshots temporary dir" into stable/havana
Make test_poll_volume_usage_with_data more reliable

This test includes 2 calls to time.time().  It assumes that the result
from the second call may be greater than the first.  This isn't
definietly true and we see periodic failures of this test in the gate as
a result.

Change the test to not worry about the specific time value that gets
set.  We set _last_vol_usage_poll to 0 before calling the function under
test.  We really only need to check that it got set to some value after
calling the function under test.  This tweak should make the test pass
consistently.

Change-Id: I6c01d2958cf671c29622562689a15ca1ddea0a3c
Closes-bug: #1270693
(cherry picked from commit 7e18d67cc9c96a14e0ca24a41820441e41457738)
Re-write sqlite BigInteger mapping test

This test makes sure that our hook that compiles BigInteger() into
INTEGER for sqlite instead of BIGINT is working as we expect.
Unfortunately, we've seen some problems running the sqlite command to
check the schema.

This patch rewrites the test to only use sqlalchemy APIs and avoid
calling out to the sqlite command for verification.

Change-Id: Id3964d0dd5b0ddf08ab745d6878840ff06b2f0dd
Closes-bug: #1271331
(cherry picked from commit 74ade48634e2b72ec0b09792e97ac77dfb5fd999)
Make test_different_fname_concurrency less racy

This test case makes a bad assumption about when a greenthread will run
and signal its done event.  The test code yields once using a sleep and
expects that the done event will get signaled before the test code runs
again.  This may not be the case.  Instead, yield up to 10 times waiting
for the done event to occur.

I was not able to reproduce this failure locally, but we're seeing it
regularly in CI.  It seems that this should resolve it.

Change-Id: I978ae9eef5bb1ac296055dc1613406e4284a625f
Closes-bug: #1270654
(cherry picked from commit a35776d337493b14d2937f0f4a03aa4b119d018b)
Merge "Make test_poll_volume_usage_with_data more reliable" into stable/havana
Merge "Re-write sqlite BigInteger mapping test" into stable/havana
Merge "Make test_different_fname_concurrency less racy" into stable/havana
Fix bug for neutron network-name

There may be cases when the network-name is not updated correctly
due to the requested networks not being owned by the tenant.

In the case when there is no matched network we use the details
from the used port.

A failed deletion of a port will raise an exception. If the
port is not found no exception will be raised.

Co-authored-by: Evgeny Fedoruk <EvgenyF@Radware.com>

Closes-Bug: #1246258
(cherry picked from commit b12da559b6a40fba4e4431d74372b5e350047525)

Conflicts:
	nova/tests/network/test_neutronv2.py

Change-Id: Ie28e88ec9a9c7a180410ae5e57f84b1f653bbffc
libvirt: Fix root disk leak in live mig

This patch makes sure that _create_images_and_backing method of the
libvirt driver (called in several places, but most problematic one is
the call in the pre_live_migration method) creates all the files the
instance needs that are not present.

Prioir to this patch - the method would only attempt to download the
image, and if it did so with the path of the ephemeral drives, it could
expose the image to other users as an ephemeral devices. See the related
bug for more detaiis.

After this patch - we properly distinguish between image, ephemeral and
swap files, and make sure that the imagebackend does the correct thing.

Closes-bug: #1251590

Co-authored-by: Loganathan Parthipan <parthipan@hp.com>

(cherry picked from commit c69a619668b5f44e94a8fe1a23f3d887ba2834d7)

Conflicts:
	nova/virt/libvirt/driver.py
Change-Id: I78aa2f4243899db4f4941e77014a7e18e27fc63e
support stevedore >= 0.14

the previous mocks made some very specific assumptions about the
behavior of an internal private method in stevedore, which changed
in 0.14. Make the mocks more flexible so they work regardless.

Change-Id: I5063c652c705fd512f90ff3897a4c590f7ba7c02
(cherry picked from commit 3992cfa3463fa7ab72a746cae38f7e212ad1a00a)