Improve performance for checking hosts AZs

When updating aggregate metadata or when adding a host into aggregate,
a check is done to make sure that hosts can't be in multiple AZs.

In order to do the check, the routine was looping over each host to look
in the DB which AZs the host is part of.
As it's quite hungry in terms of DB accesses, here the proposal is to
extend the helper method for getting all AZs by also having their hosts,
and so only do memory iterations.

Partial-Bug: #1277230

Change-Id: I7c5bc980acb17f0a0e1f8c1403dac66c305866fb
Properly skip coreutils readlink tests

Some tests in VirtDiskVFSLocalFSTestPaths class are skipped
if we're running on OSX by doing a check using is_osx() function
from tests.utils. However, these tests actually require readlink
from coreutils, so replace OS X check by more general check
for coreutils readlink presence.

This change allows to run a complete test suite on OSes rather than
Linux and OSX, such as FreeBSD.

These tests were the only consumer of tests.utils.is_osx, but
it's kept because it could be still potentially useful.

Change-Id: I7593560b84bda3719de811fbc214b1f52e1a121f
Adds IVS unit tests for new VIF firewall logic

Adds new unit tests for the IVS VIF driver to
cover the various possible plugging behaviors
added by the vif_details based logic introduced
in change I3bbcfc67036ab7389c82720add0bc0fc627bfee0.

Closes-Bug: #1301087
Change-Id: I7f0a1460b948a852c2d61d24217bbf27a4d85859
Ensure changes to api.QUOTA_SYNC_FUNCTIONS are restored.

Add a local callable restore_sync_functions() that is
established as an addCleanup() fixture callable,
making use of a copy of the QUOTA_SYNC_FUNCTIONS
dictionary to restore its previous contents in-place.

Change-Id: I7fa69ea91e517121aba0ee22a71b0594591ae308
Closes-Bug: #1329482
libvirt: Don't pass None for image_meta parameter in tests

The get_guest_config() method in the libvirt driver expects to
have a non-None value for image_meta - the to_xml() caller
will ensure this is non-None. The test cases, however, are often
calling get_guest_config() directly instead of via to_xml() and
are passing None for image_meta. Fortunately this hasn't been a
problem historically but will become one as more of the XML code
requires access to the image_meta object. Ensure the test cases
all pass {} instead of None to addres this.

Blueprint: virt-driver-vcpu-topology
Change-Id: I30fc23ef1344fcdcfef10113c6679025f60f2036
Migrate nvp-qos to generic name qos-queue

In Juno, the nvp-qos extension was renamed to qos-queue in favor of having
a generic name. In order to maintain backwards compatibility neutron
advertises both nvp-qos and qos-queue (since icehouse). This patch phases out
nvp-qos from nova so that it uses qos-queue so that in Kxxx we'll be able to
drop nvp-qos from neutron completely.

In addition this patch moves qos-queue to a constant like the other extensions
and improves a doc string.

Change-Id: Id43ea931c995865ddc3455abfb625d86dd93e78f
Closes-bug: 1329075
libvirt: Use VIR_DOMAIN_AFFECT_LIVE for paused instances

When a volume is attached to a paused instance, it does not
appear in the instance's block devices list (i.e. lsblk)
after the instance has resumed. A similar situation
happens when detaching a volume from a paused instance; the
block device continues to exists in the block devices list,
where it should not.  It was found that the volume is only
persisted in the domain's config settings and not its active
settings, since only the VIR_DOMAIN_AFFECT_CONFIG flag was used
on attach and detach volume.  In order to affect the active
settings, the VIR_DOMAIN_AFFECT_LIVE flag has to be added when
attaching and detaching volumes.

Change-Id: I9c90a410a7ecb91f5a4de28acee21fe7da49242c
Closes-Bug: #1242366
Related-Bug: #1299331
Downgrade log level when create network failed

We raised exception when create network failed because of exceed
quota, downgrade to debug level for those exceptions because
LOG.exception will generate a stacktrace.

Change-Id: I22446f08ee078c624970f233875874a67ea423b9
Catch ProcessExecutionError in revoke_cert

Catch processExecutionError if revoking the
certificate fails.

This change has been abandoned by Chuck Short, so
I am continuing where he left off (mattoliverau).

Continues abandoned change: 17741

Change-Id: I9714ea8cece87256ff5f9a936286c1da3d628af9
Closes-Bug: #883320
Co-Authored-By: chuck.short@canonical.com
Add missing stats info to BaremetalNodeState

This is the first of two patches to prepare baremetal for the removal
of capabilities in Nova HostManager.

BaremetalHostManager used to rely on capabilities info to decide which
host state class to use, VM or baremetal. This has been broken since
host capabilities reporting were removed in Nova, resulting in Nova
HostState class always being used.

BaremetalNodeState lacks stats info needed by some filters during
scheduling. This may result in failures like ComputeCapabilitiesFilter
not returning any hosts. This must be fixed before BaremetalNodeState
can be used again.

Change-Id: I63d992371640b2fd33d35eae72201fd41944f8a4
Partial-Bug: #1260265
Fix BaremetalHostManager node detection logic

This is the second patch to prepare baremetal for the removal of
capabilities in Nova HostManager.

BaremetalHostManager used to rely on capabilities info to decide which
host state class to use, VM or baremetal. This has been broken since
host capabilities reporting were removed in Nova, resulting in Nova
HostState class always being used.

This patch fixes the detection logic in BaremetalHostManager to
correctly choose BaremetalHostState class for use with baremetal nodes.
Tests are added to verify that the correct class is used for both vm
and baremetal nodes.

In reality, the new detection logic will not start to deliver the
correct HostState class until the Nova HostManager change that follows
has landed.

Change-Id: I0c2e6d985fb29f35b823cdf9a83fead2fa45082a
Partial-Bug: #1260265
Remove traces of now unused host capabilities from scheduler

This is take two of this patch. This time after preparing Baremetal
and Ironic code to work with the HostManager changes introduced in
this patch.

Host capabilities updates are no longer used by the scheduler. All host
state is instead fetched from the DB. Some left over fields are still
kept by the host manager but contain no data. This change remove the
last traces of host capabilities from the scheduler.

Baremetal/Ironic host managers used to rely on capabilities info to
decide which host state class to use, VM or baremetal. This has been
broken since host capabilities reporting were removed. This change
works in tandem with prior updates to Baremetal/Ironic node detection
logic to fix this problem.

Important: This must not be approved until the Ironic patch in
https://review.openstack.org/97447 has first landed!

Change-Id: Id5f88c0aa1df9152ddb62904b0544801a8bcfd2a
Closes-Bug: #1260265
XenAPI: disable/enable host will be failed when using XenServer

When disable/enable host with XenServer, the api of set_host_enabled()
in XenServer driver needs to get service info for the target host, but
the rpc call of set_host_enabled() did not transfer target host as
parameter, this will cause the api call failed.

This patch was let the API of set_host_enabled() use CONF.host.

Closes-Bug: #1251943

Change-Id: I2b6427df80ee3188d6f7c4eeb4d2e7be300f0126
Drop support for conductor 1.x rpc interface

In commit 330ce897c14f66a39426e716f02a620453035c15, a part of
Icehouse, we added support for the 2.x interface.  Now that Juno
development is open, we can drop support for the old API, 1.x.

For more information on this change, see:

    https://wiki.openstack.org/wiki/RpcMajorVersionUpdates

UpgradeImpact

Change-Id: Id7e605d493a53eaaa2a9afa0c99ac721f590ca3f
Drop support for scheduler 2.x rpc interface

In commit 741661a6695ccc56e5f50c4d85954cd3ed9ebda5, a part of
Icehouse, we added support for the 3.x interface.  Now that Juno
development is open, we can drop support for the old 2.x API.

For more information on this change, see:

    https://wiki.openstack.org/wiki/RpcMajorVersionUpdates

UpgradeImpact

Change-Id: I7bfad51ae46e999b30e003a5a5c7e341461dba02
Fix Flavor object extra_specs and projects handling

The Flavor.save() method was not properly handling extra_specs
and projects when deletions occur, and was doing a read/edit/save
cycle, which could involve some stale data.

This patch keeps the object-y interface to adding/removing things
from those lists, but ensures that they're added/removed individually
on the backend. The save() method is strange on Flavor, in that only
these two sets of data are actually mutable. Because of this, it's
fairly convenient to make that method no longer remotable and
break the actions into two new remotable methods that take fine-
grained lists of add/remove tasks.

Note that Flavor was not used over RPC in any previous release,
so the remotability change doesn't really affect anything. However,
if a client were to issue a remoted save() call, it would still work
and would still have the same (broken) behavior as before, where
deletes don't propagate and we just always add/update every key.

Related to blueprint compute-manager-objects-juno

Change-Id: Ic4439e05b9c0b8274e54768cbd7949aced45cc69
Don't store duplicate policies for server_group

It doesn't make sense to store same policies in a server_group.
We need to deny this usage in server_group creation.

Change-Id: I4f3ad544aef78cbbc076c7a47cca04832a2f5b4b
Closes-Bug: #1324348
Should not delete active snapshot when instance is terminated

The instance might be deleted when doing snapshot, need to check
whether the snapshot is active state before delete it.

Change-Id: Ic12b8dfd05ad7f36831dd073cef746f094d8f2d0
Closes-Bug: #1329882
Add new fields to the networks table

Adds a migration to add mtu, dhcp_server, enable_dhcp, and
share_address to the networks table. The field for dhcp_server defaults
to gateway, since gateway is currently used to determine the
dhcp_server value. The value for mtu and share address is determined by
a flag, so copy the value from the flag into the table during the
migration.

Partially-implements blueprint better-support-for-multiple-networks

Change-Id: I8278fb125e6e478b519c875eca9ebbf778d88d74
Adds network fields to object

Adds mtu, dhcp_server, enable_dhcp, and share_address to network
objects and provide data migration on save.

Partially-implements blueprint better-support-for-multiple-networks

Change-Id: I318791a9d7f53534b3383221cdfb7dfe63dd116b