Improve performance for checking hosts AZs

When updating aggregate metadata or when adding a host into aggregate,
a check is done to make sure that hosts can't be in multiple AZs.

In order to do the check, the routine was looping over each host to look
in the DB which AZs the host is part of.
As it's quite hungry in terms of DB accesses, here the proposal is to
extend the helper method for getting all AZs by also having their hosts,
and so only do memory iterations.

Partial-Bug: #1277230

Change-Id: I7c5bc980acb17f0a0e1f8c1403dac66c305866fb
Accurate exception info in api layer for aggregate

Adjust log information for exceptions in api layer for aggregate
according to
https://wiki.openstack.org/wiki/LoggingStandards#INFO_log_level_is_for_Operators
the LOG.info was removed because nova api already reported error to operator

Change-Id: I89394407495cf2d346e1f532ada236a4364aab24
VMware: deprecate ESX driver from virt configuration

Remove the VMware ESX driver configuration from the virt driver
configuration help options.

The commit d041b95b8977e77a51e6031c38f0011b44d3266a marked the
VMware ESX driver as deprecated. This is the first step in
starting to remove the driver in Juno.

Change-Id: I955ffee336feeca68994d9ea4e244f3c7e10eed2
Properly skip coreutils readlink tests

Some tests in VirtDiskVFSLocalFSTestPaths class are skipped
if we're running on OSX by doing a check using is_osx() function
from tests.utils. However, these tests actually require readlink
from coreutils, so replace OS X check by more general check
for coreutils readlink presence.

This change allows to run a complete test suite on OSes rather than
Linux and OSX, such as FreeBSD.

These tests were the only consumer of tests.utils.is_osx, but
it's kept because it could be still potentially useful.

Change-Id: I7593560b84bda3719de811fbc214b1f52e1a121f
Fix EC2 not found errors for volumes and snapshots.

According to AWS docs
(http://docs.aws.amazon.com/AWSEC2/latest/APIReference/
api-error-codes.html)

Change-Id: I43af230b975263bb2a14e0cc0bd8718f55bd8a55
Closes-Bug: #1323813
Adds IVS unit tests for new VIF firewall logic

Adds new unit tests for the IVS VIF driver to
cover the various possible plugging behaviors
added by the vif_details based logic introduced
in change I3bbcfc67036ab7389c82720add0bc0fc627bfee0.

Closes-Bug: #1301087
Change-Id: I7f0a1460b948a852c2d61d24217bbf27a4d85859
VMware: spawn refactor - DatastorePath class

Part of spawn refactor work to consolidate a lot of the datastore path
manipulation code. The DatastorePath class is a simpler version of a
class of the same name in earlier versions of
https://review.openstack.org/#/c/87002, and is intentionally limited in
the ways that DatastorePath objects can be constructed. No attempts are
made to ensure that the datastore, directories and files referenced in
these objects are valid on the remote system.

partial blueprint vmware-spawn-refactor

Change-Id: Id65d0308d32835048229e7a3d98b84af2b13583c
VMware: remove dsutil.split_datastore_path

The functionality of split_datatore_path has been subsumed by
the DatastorePath utility class, so remove it and convert
all callers to use the latter.

Change-Id: I07a2cfe15fccaca4aee25bc43157e6c604b91831
VMware: spawn refactor - Datastore class

Introduces the start of a simple Datastore class to encapsulate relevant
data of a datastore.

partial blueprint vmware-spawn-refactor

Change-Id: Ic0ee6cdd47e7cbd66cf456776c61d0b85fb51a0a
Disable rescheduling instance when no retry info

When scheduler_max_attempts=1 means disable rescheduling, compute
won't reschedule instance before. But build_and_run_instance didn't
hornor this behavior. And cause nova reschedule instance infinitely.
This patch fix this problem, make build_and_run_instance won't reschedule
instance without retry info.

Change-Id: I52d6569c1731b5fd5a10f39ff04a40b43d3857a1
Partial-Bug: #1326207
Ensure changes to api.QUOTA_SYNC_FUNCTIONS are restored.

Add a local callable restore_sync_functions() that is
established as an addCleanup() fixture callable,
making use of a copy of the QUOTA_SYNC_FUNCTIONS
dictionary to restore its previous contents in-place.

Change-Id: I7fa69ea91e517121aba0ee22a71b0594591ae308
Closes-Bug: #1329482
libvirt: Don't pass None for image_meta parameter in tests

The get_guest_config() method in the libvirt driver expects to
have a non-None value for image_meta - the to_xml() caller
will ensure this is non-None. The test cases, however, are often
calling get_guest_config() directly instead of via to_xml() and
are passing None for image_meta. Fortunately this hasn't been a
problem historically but will become one as more of the XML code
requires access to the image_meta object. Ensure the test cases
all pass {} instead of None to addres this.

Blueprint: virt-driver-vcpu-topology
Change-Id: I30fc23ef1344fcdcfef10113c6679025f60f2036
Remove duplicate code in Objects create() function

Remove duplicate code updates.pop('id', None) in create() function.
Because if self.obj_attr_is_set('id'):
            raise exception.ObjectActionError(action='create',
            reason='already created')
already make sure object doesn't have 'id' values, which means that
'id' won't appear in updates.

Change-Id: I582ef8857b32c07ea8ba1eb5fc1e64516110f3ce
Handle service creation race by service workers

This handle a race where a sibling has already created the service
record in the database.  This fix is similar to that of commit
f6c341b4.

Change-Id: I7975fc632eee34390c5d7dc275b52363ad45475e
Closes-bug: 1326901
Migrate nvp-qos to generic name qos-queue

In Juno, the nvp-qos extension was renamed to qos-queue in favor of having
a generic name. In order to maintain backwards compatibility neutron
advertises both nvp-qos and qos-queue (since icehouse). This patch phases out
nvp-qos from nova so that it uses qos-queue so that in Kxxx we'll be able to
drop nvp-qos from neutron completely.

In addition this patch moves qos-queue to a constant like the other extensions
and improves a doc string.

Change-Id: Id43ea931c995865ddc3455abfb625d86dd93e78f
Closes-bug: 1329075
libvirt: Use VIR_DOMAIN_AFFECT_LIVE for paused instances

When a volume is attached to a paused instance, it does not
appear in the instance's block devices list (i.e. lsblk)
after the instance has resumed. A similar situation
happens when detaching a volume from a paused instance; the
block device continues to exists in the block devices list,
where it should not.  It was found that the volume is only
persisted in the domain's config settings and not its active
settings, since only the VIR_DOMAIN_AFFECT_CONFIG flag was used
on attach and detach volume.  In order to affect the active
settings, the VIR_DOMAIN_AFFECT_LIVE flag has to be added when
attaching and detaching volumes.

Change-Id: I9c90a410a7ecb91f5a4de28acee21fe7da49242c
Closes-Bug: #1242366
Related-Bug: #1299331
Downgrade log level when create network failed

We raised exception when create network failed because of exceed
quota, downgrade to debug level for those exceptions because
LOG.exception will generate a stacktrace.

Change-Id: I22446f08ee078c624970f233875874a67ea423b9
Sync oslo lockutils to nova

There is some update for lockutils in oslo, sync from oslo

The commit to be merged:
edd73c9 Merge "Improve help strings"
e446fe5 Merge "Fixed several typos"
fe3389e Improve help strings
f3f14c9 Fixed several typos
0f495ee Emit a log statement when releasing internal lock
f0dd798 Remove rendundant parentheses of cfg help strings
006d9a2 Allow external locks to work with threads
9b73c19 Re-enable file-based locking behavior
edce46c Use Posix IPC in lockutils
ac84a40 Update log translation domains
ee4df7f Merge "Add remove external lock files API in lockutils"
fcf517d Update oslo log messages with translation domains
37a1de8 Move the released file lock to the successful path
3c125e6 Merge "Use threading.ThreadError instead of reraising IOError"
de826f2 Merge "Have the interprocess lock follow lock conventions"
b0d0c33 Add remove external lock files API in lockutils
e71cd1a Merge "Trivial: Make vertical white space after license header consistent"
4f6190a Use threading.ThreadError instead of reraising IOError
195f0b1 Have the interprocess lock follow lock conventions
15cca4b lockutils: move directory creation in lock class
81fe39e lockutils: remove lock_path parameter
14d3669 lockutils: expand add_prefix
dc06d55 lockutils: do not grab the lock in creators
a0948cb lockutils: remove local usage
df8e051 lockutils: split code handling internal/external lock
8b2b0b7 Use hacking import_exceptions for gettextutils._
6d0a6c3 Correct invalid docstrings
6fa29ae Trivial: Make vertical white space after license header consistent
12bcdb7 Remove vim header
79e6bc6 fix lockutils.lock() to make it thread-safe
ace5120 Add main() to lockutils that creates temp dir for locks

Related-bug: #1256306

Change-Id: Iced751f059bd9a893ccefb5f98d63df29f40c622
Catch ProcessExecutionError in revoke_cert

Catch processExecutionError if revoking the
certificate fails.

This change has been abandoned by Chuck Short, so
I am continuing where he left off (mattoliverau).

Continues abandoned change: 17741

Change-Id: I9714ea8cece87256ff5f9a936286c1da3d628af9
Closes-Bug: #883320
Co-Authored-By: chuck.short@canonical.com
Add missing stats info to BaremetalNodeState

This is the first of two patches to prepare baremetal for the removal
of capabilities in Nova HostManager.

BaremetalHostManager used to rely on capabilities info to decide which
host state class to use, VM or baremetal. This has been broken since
host capabilities reporting were removed in Nova, resulting in Nova
HostState class always being used.

BaremetalNodeState lacks stats info needed by some filters during
scheduling. This may result in failures like ComputeCapabilitiesFilter
not returning any hosts. This must be fixed before BaremetalNodeState
can be used again.

Change-Id: I63d992371640b2fd33d35eae72201fd41944f8a4
Partial-Bug: #1260265