Add log_handler to implement the publish_errors config option

Synced from Olso.
Commit Id is Ibf28ba17f81b5ddeda653f6a3501ecac18ffcd42

This handler sends all error logs as rpc notifications.
Note: when using publish_errors you also need to set the
notification_driver option too.

Change-Id: Ic97b9c8b84d51c84a53ace5b86cc2e72c744f136
Closes-bug: #1240349
(cherry picked from commit 1c4e279e187d1b864680c9be7849130cb7f12801)
Fix `NoopQuotaDriver.get_(project|user)_quotas` format

The quota API extension expects `get_project_quotas` and `get_user_quotas` to
return a dictionary where the value is another dictionary with a `limit` key.

The `DbQuotaDriver` adhered to this spec, but the `NoopQuotaDriver` didn't.

This fixes the `NoopQuotaDriver` to return the results in the correct format.

Fixes bug 1244842

Change-Id: Iea274dab1c3f10c3cb0a2815f431e15b4d4934b1
(cherry picked from commit 711a12b4029cd1544d26d147d8a67e110e056124)
Fix interface-attach removes existing interfaces from db

The following commit 394c693e359ed4f19cc2f7d975b1f9ee5500b7f6 changed
allocate_port_for_instance() to only return the ports that were created
rather than all of the ports on the instance which broke the attach-interface
code.

This patch fixes this issue by remove the sync decorators from:
allocate_for_instance, allocate_port_for_instance, and
deallocate_port_for_instance to just _build_instance_nw_info which
is called in all these cases instead.

Closes-bug: #1223859

(cherry picked from commit 1957339df302e2da75e0dbe78b5d566194ab2c08)

Conflicts:
	nova/network/neutronv2/api.py

Change-Id: I66eb0c0ab926e0a8d1e2c9cfe1f7fd579ea3aa27
libvirt: Allow delete to complete when a volume disconnect fails

If an instance which was booted from a volume fails to spawn, the
instance is left in an ERROR state.  If the failure is in the networking
stage Nova will not have attached the volume via the Cinder driver,
and so the volume remains in an available state.  In this state the
volume may be deleted or assigned to another instance.
During the subsequent delete an exception may be thrown from
the cinder driver which prevents the instance from completing the
deletion process leaving the instance stuck in Error and unable to
be deleted.

Volume exceptions should be logged and ignored when the instance
is being deleted.

Closes-Bug #1222979

(cherry picked from commit 506a8f58cf4b8cecf90b647c7deba47da2a4dfec)
Conflicts:
	nova/tests/virt/libvirt/test_libvirt.py

Change-Id: Icb3796b0ddba25cf344953a649b2e762fab6d782
Fix performance of Server List with Neutron for Admins

https://review.openstack.org/#/c/30048/ optimized the
SecurityGroupsOutputController for the case where the server
list only contains one server, but in all other cases the current
code calls the Neutron driver in a way that makes it retrieve
all ports and security groups visible to the tenant.

For users with a Neutron admin role this retrieves all ports
and all SecGroups in the system, which on a large system is a
major performance issue and often leads to client timeouts.
Normally these users have further qualified their query to a
specific tenant or host, or are maybe just trying to get their
own list of servers.

If instead we pass the pre-filtered list of servers to Neutron
it can query for ports associated with just the device IDs of
the servers.  We then use that list of ports to build a
specific list of security groups IDs to query for.

This approach means that we can remove the initial
optimization for a single server case.

Closes-Bug: 1228384

(cherry picked from commit 18c3ac4a8935b9997dbb55181d5dbb5232ef2c27)

Conflicts:
	nova/tests/api/openstack/compute/contrib/test_neutron_security_groups.py

Change-Id: I6f72533056e4f336f7578cc883fd1a125c2048a9
VMware: fix bug when more than one datacenter exists

In the case that there was more than one datacenter defined on the VC,
then spawning an instance would result in an exception. The reason for this
was that the nova compute would not set the correct datacenter for the
selected datastore.

The fix also takes care of the correct folder selection. This too was a
result of not selecting the correct folder for the data center.

The 'fake' configuration was updated to contain an additional data
center with its on datastore.

Closes-Bug: #1180044
Closes-Bug: #1214850

Co-authored-by: Shawn Harsock <hartsocks@vmware.com>

(cherry picked from commit a25b2ac5f440f7ace4678b21ada6ebf5ce5dff3c)

Conflicts:

	nova/tests/virt/vmwareapi/test_vmwareapi.py
	nova/virt/vmwareapi/fake.py

Change-Id: Ib61811fffcbc80385efc3166c9e366fdaa6432bd
Fix a lazy-load exception in security_group_update()

When security_group_api is nova, the update_security_group api raises http
500 error: group rules fails with lazy load exception. Because the rule
infos of the security group will be return in response body, but the rule
attribute hasn't been preloaded.

Closes-Bug: #1244018
(cherry picked from commit b9b55935cb61cc1726525b1c4963e352a3a81ea9)

Conflicts:
	nova/tests/api/openstack/compute/contrib/test_security_groups.py
Change-Id: Ie07879ac22e4add90f75968778bf334915e8349c
libvirt: Fix infinite loop waiting for block job

Libvirt may return an empty dict when a block job has completed.
In this case, _wait_for_block_job will return True, causing
its callers to loop.

Affects volume_snapshot_delete, swap_volume, and live_snapshot.

(See http://libvirt.org/git/?p=libvirt.git;a=commit;h=0f9e67bf )

Closes-Bug: #1260123
Change-Id: Iab81299f0ce32a14e46aee0fd389ed29f7e503b8
(cherry picked from commit e3ac20fca7494122a4c0ed2edc00377aa92cb49d)
VMware: unnecessary session reconnection

An error FAULT_NOT_AUTHENTICATED could imply that there is a problem
with the backend or that no values were found (for example there are
no running VMS). In the latter case the connection should not
be restarted.

The patch also ensures that the session is restarted if there are
connection issues.

This issue occasionally causes CI issues with minesweeper.

Closes-bug: #1265512

Co-authored-by: Shawn Harsock <hartsocks@vmware.com>

Conflicts:

	nova/tests/virt/vmwareapi/test_vmwareapi.py

Change-Id: Ibfa83d0b5b93680890aad8c88c041fa344765448
(cherry picked from commit 55296828a7d7a03736a21d846da15571ca22ec9e)
Fix bug for neutron network-name

There may be cases when the network-name is not updated correctly
due to the requested networks not being owned by the tenant.

In the case when there is no matched network we use the details
from the used port.

A failed deletion of a port will raise an exception. If the
port is not found no exception will be raised.

Co-authored-by: Evgeny Fedoruk <EvgenyF@Radware.com>

Closes-Bug: #1246258
(cherry picked from commit b12da559b6a40fba4e4431d74372b5e350047525)

Conflicts:
	nova/tests/network/test_neutronv2.py

Change-Id: Ie28e88ec9a9c7a180410ae5e57f84b1f653bbffc
libvirt: Fix root disk leak in live mig

This patch makes sure that _create_images_and_backing method of the
libvirt driver (called in several places, but most problematic one is
the call in the pre_live_migration method) creates all the files the
instance needs that are not present.

Prioir to this patch - the method would only attempt to download the
image, and if it did so with the path of the ephemeral drives, it could
expose the image to other users as an ephemeral devices. See the related
bug for more detaiis.

After this patch - we properly distinguish between image, ephemeral and
swap files, and make sure that the imagebackend does the correct thing.

Closes-bug: #1251590

Co-authored-by: Loganathan Parthipan <parthipan@hp.com>

(cherry picked from commit c69a619668b5f44e94a8fe1a23f3d887ba2834d7)

Conflicts:
	nova/virt/libvirt/driver.py
Change-Id: I78aa2f4243899db4f4941e77014a7e18e27fc63e
Merge "VMware: unnecessary session reconnection" into stable/havana
Merge "Fix a lazy-load exception in security_group_update()" into stable/havana
Merge "libvirt: Fix infinite loop waiting for block job" into stable/havana
Merge "Add log_handler to implement the publish_errors config option" into stable/havana
Merge "libvirt: Fix root disk leak in live mig" into stable/havana
Merge "Fix performance of Server List with Neutron for Admins" into stable/havana
Merge "Fix bug for neutron network-name" into stable/havana
Setup destination disk from virt_disk_size

When running live-migration --block-migrate on a qcow2 backed
VM without cow image, the destination qcow2 file should be created
with the virtual disk size. For raw images, the virt_disk_size
is set to disk_size to ensure that virt_disk_size is always the
size of the disk that should be re-created.

Update unit tests to be more strict and check for sizes to be correct.

Closes-Bug: #1257355

Change-Id: Ie3be46024f06b9f59af92f5e3918a1958386d4f1
Merge "libvirt: Allow delete to complete when a volume disconnect fails" into stable/havana