Improve performance for checking hosts AZs

When updating aggregate metadata or when adding a host into aggregate,
a check is done to make sure that hosts can't be in multiple AZs.

In order to do the check, the routine was looping over each host to look
in the DB which AZs the host is part of.
As it's quite hungry in terms of DB accesses, here the proposal is to
extend the helper method for getting all AZs by also having their hosts,
and so only do memory iterations.

Partial-Bug: #1277230

Change-Id: I7c5bc980acb17f0a0e1f8c1403dac66c305866fb
Properly skip coreutils readlink tests

Some tests in VirtDiskVFSLocalFSTestPaths class are skipped
if we're running on OSX by doing a check using is_osx() function
from tests.utils. However, these tests actually require readlink
from coreutils, so replace OS X check by more general check
for coreutils readlink presence.

This change allows to run a complete test suite on OSes rather than
Linux and OSX, such as FreeBSD.

These tests were the only consumer of tests.utils.is_osx, but
it's kept because it could be still potentially useful.

Change-Id: I7593560b84bda3719de811fbc214b1f52e1a121f
Allow user to specify image to use during rescue - V3 API changes

This commit includes the V3 changes.
Making changes in V3 API to take an extra parameter rescue_image_ref
as part of the rescue action API.
If specified, this image_ref will be used to rescue the instance.
If the image is not specified, then the base image ref is used by
default.

Partially Implements: blueprint allow-image-to-be-specified-during-rescue

DocImpact

Change-Id: I7a9d88bf4dfe73879247f5b82a52928fdde76510
libvirt: return the correct instance path while cleanup_resize

If we resized a pre-grizzly instance with grizzly or later nova
to another host, while the resize confirmation process,
_cleanup_resize will find the instance resize backup dir and
delete it, but a wrong xxx_resize dir like ${uuid}_resize,
instead of the correct ${name}_resize will be found.
This is because the instance is a resized one which original
instance dir exists on another host(the dest host),
get_instance_path method could not find the original instance
dir on the source host, so the path with uuid will be returned,
and the `target` existing check in _cleanup_resize is failed,
then the ${name}_resize dir will never be deleted.

Closes-bug: #1290294
Change-Id: I904b6751dec740e001f5ec29f637ef456528746f
libvirt: Remove use of db for flavor extra specs in tests

The watchdog tests were using the, now discouraged, approach of calling
db.flavor_extra_specs_update_or_create to set flavour extra specs for
the test case. Rewrite the test cases to mock the Flavor.get_by_id
method to return a fake flavor object.

Change-Id: I92b45762a8f079b0e7bef7f9daf4cf5ada78896f
libvirt: Don't pass None for image_meta parameter in tests

The get_guest_config() method in the libvirt driver expects to
have a non-None value for image_meta - the to_xml() caller
will ensure this is non-None. The test cases, however, are often
calling get_guest_config() directly instead of via to_xml() and
are passing None for image_meta. Fortunately this hasn't been a
problem historically but will become one as more of the XML code
requires access to the image_meta object. Ensure the test cases
all pass {} instead of None to addres this.

Blueprint: virt-driver-vcpu-topology
Change-Id: I30fc23ef1344fcdcfef10113c6679025f60f2036
Migrate nvp-qos to generic name qos-queue

In Juno, the nvp-qos extension was renamed to qos-queue in favor of having
a generic name. In order to maintain backwards compatibility neutron
advertises both nvp-qos and qos-queue (since icehouse). This patch phases out
nvp-qos from nova so that it uses qos-queue so that in Kxxx we'll be able to
drop nvp-qos from neutron completely.

In addition this patch moves qos-queue to a constant like the other extensions
and improves a doc string.

Change-Id: Id43ea931c995865ddc3455abfb625d86dd93e78f
Closes-bug: 1329075
Deprecate glance_* configuration settings

Create a new section in the configuration file called 'glance'.
Move all of the glance_* configuration settings to this section.

DocImpact

The table below has the changes:

+---------------------------------+----------------------------+
| 'DEFAULT' Section               | 'glance' Section           |
|---------------------------------|----------------------------|
| glance_host                     | host                       |
| glance_port                     | port                       |
| glance_protocol                 | protocol                   |
| glance_api_servers              | api_servers                |
| glance_api_insecure             | api_insecure               |
| glance_num_retries              | num_retries                |
| allowed_direct_url_schemes      | allowed_direct_url_schemes |
+---------------------------------+----------------------------+

UpgradeImpact
The changes are backward compatible.

Change-Id: I829cf006f7e7c53a36f5e12b40f60f875fe8f331
Fix Flavor object extra_specs and projects handling

The Flavor.save() method was not properly handling extra_specs
and projects when deletions occur, and was doing a read/edit/save
cycle, which could involve some stale data.

This patch keeps the object-y interface to adding/removing things
from those lists, but ensures that they're added/removed individually
on the backend. The save() method is strange on Flavor, in that only
these two sets of data are actually mutable. Because of this, it's
fairly convenient to make that method no longer remotable and
break the actions into two new remotable methods that take fine-
grained lists of add/remove tasks.

Note that Flavor was not used over RPC in any previous release,
so the remotability change doesn't really affect anything. However,
if a client were to issue a remoted save() call, it would still work
and would still have the same (broken) behavior as before, where
deletes don't propagate and we just always add/update every key.

Related to blueprint compute-manager-objects-juno

Change-Id: Ic4439e05b9c0b8274e54768cbd7949aced45cc69
Create lvm.py module containing helper API for LVM

Currently there are some LVM API calls in 'nova/virt/libvirt/utils.py'
and also many other APIs are there.
To make a role of the code be clear, I would like to suggest to
create 'nova/virt/libvirt/lvm.py' and move APIs which are related
to LVM to lvm.py.

Change-Id: I4d80f574631d52012152c72d1db3069dd57a5550
Signed-off-by: Mitsuhiro Tanino <mitsuhiro.tanino@hds.com>
Implements: blueprint lvm-driver-for-shared-storage
Log exception if max scheduling attempts exceeded

This logs the last exception seen by the scheduler to help troubleshooting.

When an administrator shows an errored instance, the fault reported is
NoValidHost which is easily misread as insufficient capacity.

This is currently covered by tests.

Change-Id: I636ec9809ded2d4e3404c1f13e0d7635b9826bfe
Add new fields to the networks table

Adds a migration to add mtu, dhcp_server, enable_dhcp, and
share_address to the networks table. The field for dhcp_server defaults
to gateway, since gateway is currently used to determine the
dhcp_server value. The value for mtu and share address is determined by
a flag, so copy the value from the flag into the table during the
migration.

Partially-implements blueprint better-support-for-multiple-networks

Change-Id: I8278fb125e6e478b519c875eca9ebbf778d88d74
Adds network fields to object

Adds mtu, dhcp_server, enable_dhcp, and share_address to network
objects and provide data migration on save.

Partially-implements blueprint better-support-for-multiple-networks

Change-Id: I318791a9d7f53534b3383221cdfb7dfe63dd116b
Don't replace instance object with dict in _allocate_network()

The compute manager _instance_update() method returns a dict
version of the modified instance.  allocate_network() is the only
method which uses this return value rather than ignoring it, which
has the side effect of replaces the instance object it has been passed
with a dict.  None of methods called after this update need the modified
state values, so we can just ignore the return value as all of the
other calls do.

Passing on an object instead of a dict allows us to clean up the
system_metadata handling in _alloc_network_async() as well.

Related to blueprint compute-manager-objects-juno

Change-Id: I3249bba03edab4dd45eec72f7abdb15d7800346e
wrap_instance_event() shouldn't swallow return codes

The wrap_instance_event() function, when used as a decorator,
will currently swallow return codes.  This prevents it from
being used on a function that returns a value that the caller
actually cares about.

Change-Id: I72616b0afcd3ab8fae81742bf4b7754cd959ef3a
Closes-Bug: 1311793
Merge "Deprecate glance_* configuration settings"
Merge "wrap_instance_event() shouldn't swallow return codes"
Merge "libvirt: Remove use of db for flavor extra specs in tests"
Merge "libvirt: return the correct instance path while cleanup_resize"
Merge "Allow user to specify image to use during rescue - V3 API changes"