Improve performance for checking hosts AZs

When updating aggregate metadata or when adding a host into aggregate,
a check is done to make sure that hosts can't be in multiple AZs.

In order to do the check, the routine was looping over each host to look
in the DB which AZs the host is part of.
As it's quite hungry in terms of DB accesses, here the proposal is to
extend the helper method for getting all AZs by also having their hosts,
and so only do memory iterations.

Partial-Bug: #1277230

Change-Id: I7c5bc980acb17f0a0e1f8c1403dac66c305866fb
minor corrections to devref rpc page

Correct wording to proper English and remove extra space between words while
in there.

DocImpact

Change-Id: If4b3d9d81d7193107e84867e4ab713c413a4976d
Correct exception info format of v3 flavor manage

The existing flavor manage may raise HTTPBadRequest exception,
But the msg info didn't use '_' prefix. This patch adds it.

Change-Id: I736bf4b0e78ecc0bb753d376ee162f19fafc09e1
Allow user to specify image to use during rescue - V3 API changes

This commit includes the V3 changes.
Making changes in V3 API to take an extra parameter rescue_image_ref
as part of the rescue action API.
If specified, this image_ref will be used to rescue the instance.
If the image is not specified, then the base image ref is used by
default.

Partially Implements: blueprint allow-image-to-be-specified-during-rescue

DocImpact

Change-Id: I7a9d88bf4dfe73879247f5b82a52928fdde76510
Check the length of aggregate metadata

PostgreSQL gives an error if you try to shove 256+ characters
into a 255-character field. So we need raise a HTTPBadRequest
explanation if the length of aggregate_metadata.keys or
aggregate_metadata.values is above 255.

Change-Id: I2c778f2237ba5bd2aa8335a0eae80f3aad3e9157
Closes-Bug: #1219693
Add unit tests for ipv4/ipv6 format validation

We found a bug that the other jsonschema format check('uri') did
not work for input validation. There was not any unit tests for
the format check, then we could not find it in long term.
The bug is fixed with Ibb2b7f27c374136e6865b28786d10ca1365e5709 .
And this patch adds unit tests for the format 'ipv4' and 'ipv6'.

Change-Id: I6f6636708e3f9a4eee14451f2baacb50002f1db8
Related-Bug: #1326599
Remove XML namespace from some v3 extensions

The XML support was removed from the V3 API plugins,
this XML namespace was not used again.

Change-Id: I427597dd3f6d454037a791f911756a97f343966d
Separate the PCI device object handling code

Currently the PCI device object includes several functions like
alloc/free/claim etc. However, the NovaObject should not be used
this way, and it makes the PCI device object really different
with other NovaObject implementations.

This patch duplicates the PCI device handling code as separated
functions. No logic changes but simply code copy. A later patch
will delete the code at PciDevice object.

Closes-Bug: #1273852

Change-Id: I2389d29218d8619998f8df973004af004e8b1b76
Use new pci device handling code in pci_manager

As we remove some pci device method from PCI device object,
update the pci manager accordingly.

Closes-Bug: #1273852

Change-Id: I0ec0e7b03625888a207918816b6e2d537b6d7d54
Remove the device handling from pci device object

Remove the device handling code at pci device object to make
it similar to other NovaObject.

Remove some usage in test cases.

Closes-Bug: #1273852

Change-Id: I4487ae0b36de18365d12593f480c05f0fe726769
libvirt: return the correct instance path while cleanup_resize

If we resized a pre-grizzly instance with grizzly or later nova
to another host, while the resize confirmation process,
_cleanup_resize will find the instance resize backup dir and
delete it, but a wrong xxx_resize dir like ${uuid}_resize,
instead of the correct ${name}_resize will be found.
This is because the instance is a resized one which original
instance dir exists on another host(the dest host),
get_instance_path method could not find the original instance
dir on the source host, so the path with uuid will be returned,
and the `target` existing check in _cleanup_resize is failed,
then the ${name}_resize dir will never be deleted.

Closes-bug: #1290294
Change-Id: I904b6751dec740e001f5ec29f637ef456528746f
Attach/detach interface to paused instance with affect live flag

Currently attach/detach interface to paused instance is only with
VIR_DOMAIN_AFFECT_CONFIG flags. After instance unpause, there isn't
any effect on the instance, unless restart the instance. So add
flag VIR_DOMAIN_AFFECT_LIVE when attach/detach interface.

Change-Id: Iedfc7f6fc06e72d8e3eecebede1274582c8043fd
Closes-Bug: #1299331
libvirt: Close opened file explicitly

Every opened file should be closed properly.

Change-Id: I5405638ba033f75957461c0ad33be9f0c6a17e3b
libvirt: Remove use of db for flavor extra specs in tests

The watchdog tests were using the, now discouraged, approach of calling
db.flavor_extra_specs_update_or_create to set flavour extra specs for
the test case. Rewrite the test cases to mock the Flavor.get_by_id
method to return a fake flavor object.

Change-Id: I92b45762a8f079b0e7bef7f9daf4cf5ada78896f
libvirt: Don't pass None for image_meta parameter in tests

The get_guest_config() method in the libvirt driver expects to
have a non-None value for image_meta - the to_xml() caller
will ensure this is non-None. The test cases, however, are often
calling get_guest_config() directly instead of via to_xml() and
are passing None for image_meta. Fortunately this hasn't been a
problem historically but will become one as more of the XML code
requires access to the image_meta object. Ensure the test cases
all pass {} instead of None to addres this.

Blueprint: virt-driver-vcpu-topology
Change-Id: I30fc23ef1344fcdcfef10113c6679025f60f2036
Make check_instance_shared_storage() use objects

This changes the check_instance_shared_storage () method in compute manager
to take an instance object.

The change is mostly cosmetic here to remove one more case of an instance dict
being passed by the rpcapi since the instance parameter isn't actually used on
the server end of this call anyway !

Related to blueprint compute-manager-objects-juno

Change-Id: Id22306916d5d32af978955e02ee95e991c733bab
Deprecate neutron_* configuration settings

Create a new section in the configuration file called 'neutron'.
Move all of the neutron_* configuration settings to this section.

DocImpact

The table below has the changes:

+---------------------------------+-------------------------+
| 'DEFAULT' Section               | 'neutron' Section       |
|---------------------------------|-------------------------|
| neutron_url                     | url                     |
| neutron_url_timeout             | url_timeout             |
| neutron_admin_username          | admin_username          |
| neutron_admin_password          | admin_password          |
| neutron_admin_tenant_id         | admin_tenant_id         |
| neutron_admin_tenant_name       | admin_tenant_name       |
| neutron_region_name             | region_name             |
| neutron_admin_auth_url          | admin_auth_url          |
| neutron_api_insecure            | api_insecure            |
| neutron_auth_strategy           | auth_strategy           |
| neutron_region_name             | region_name             |
| neutron_ovs_bridge              | ovs_bridge              |
| neutron_extension_sync_interval | extension_sync_interval |
| neutron_ca_certificates_file    | ca_certificates_file    |
+---------------------------------+-----------------------=-+

UpgradeImpact
The changes are backward compatible.

Change-Id: I3308be33f60c4add3e2e78b75f238059a9b3d876
Add configurable HTTP timeout to cinder API calls

Some Nova API calls require synchronous use of cinder.  (e.g. volume
list)  If the call to cinder lands on a slow performing or otherwise bad
cinder API node, the user of Nova API will see poor performance.

This patch adds the ability to control HTTP timeouts when calling the
cinder API.  By configuring a timeout, nova will be able to more quickly
retry its cinder request and hopefully land on a better performing
cinder API node.  This will help make the API response times more
predictable.

For backward compatibility, the default timeout value is None.

Closes-Bug: 1329797
DocImpact

Change-Id: I5cccced524d9a9d69ed41f755781d0b7ae3dc970
Fix doc for service list

If a service is enabled, the "disabled_reason" field in response
for /v2/.{tenant_id}./os-services/detail is null, however, currently
it's specified as "" in the doc, thus may be confusing to users.

This patch change the template and doc, it should not cause issue to
existing applications.

Change-Id: Ia71ec4c97a355bcc1a7b63e6107db77f80a5d843
Close-bug: 1328382
Add helper functions for getting local disk

Following functions are added in order to be used by validate_bdm
and resource_tracker code.
get_bdm_ephemeral_disk_size: return sum of eph disk size of bdms
get_bdm_swap_list: return a swap disk list of bdms
get_bdm_local_disk_num: return local disk number (include eph and swap)

Change-Id: I215970bea72b502042472aa310fe64ffc4304b25