Allow configuring the wsgi pool size

Expose the number of greenthreads in wsgi pool to configuration. This
allows to put a limit to the number of resources like memcache
connections used.

DocImpact
Change-Id: I6e53d6e12661ed2f42a6b543891b13eec0c8300c
Update openstack/common/periodic_task

The following commits are in this update:

1771a77 Adjust import order according to PEP8 imports rule
4bfb7a2 Apply six for metaclass
0ea069e Removed calls to locals()
1a2df89 Enable H302 hacking check

Change-Id: I5d15d9772002a2f764acef16e28bcfa16a9760c0
Fixes get_vm_storage_paths issue for Hyper-V V2 API

Add get_vm_storage_paths function to vmutilsv2.py

get_vm_storage_paths implementation on Hyper-V V2 API changed
compared to previous V1 API. In previous V1 APIs, using
Msvm_ResourceAllocationSettingData class can get storage
paths, but in V2 API, only using Msvm_StorageAllocationSettingData
class can get the storage paths.

Closes-Bug: #1248463

Change-Id: Ica73221524f162a6ffcd9dc64ee33c85fb5ad31d
convert min_count and max_count to type int in nova v3 api

this converts min_count and max_count to type:int in nova
v3 api.
If the string type hasn't been converted, in the method
_check_num_instances_quota of nova/compute/api.py, something
like req_ram = '2'*128 will occur. That's why the issue in
bug 1254616 arise.

Change-Id: I567b249e7a30681980ed760d5d09aad4c8e603b9
Closes-bug: 1254616
Fix a lazy-load exception in security_group_update()

When security_group_api is nova, the update_security_group api raises http
500 error: group rules fails with lazy load exception. Because the rule
infos of the security group will be return in response body, but the rule
attribute hasn't been preloaded.

Closes-Bug: #1244018
Change-Id: Ie07879ac22e4add90f75968778bf334915e8349c
Check Neutron port quota during validate_networks in API

Unless ports are passed into Nova it will create ports on the
requested networks as part of the network allocation in
the compute manager. However if the user exceeds their
port quota the instance will end up in an Error state, having
first been re-scheduled a number of times.

It would be much better if the quota failure was detected as
part of the network validation in the API server, so that an
error can be reported to the user and the creation failed.

A full fix would include reserving or creating the ports at
this stage, but there is no reservation mechanism in the
Neutron API, and port creation depends in some cases
on mac addresses only available on the compute manager.

Instead this change just validates the quota and adjusts the
max_count to be consistent with that quota, which doesn't
guarantee that the create will work, but does catch the
majority of cases.

Refs bug: 1172808

Change-Id: Iaaee059a6746fad68049712f94b2f8cfea6ab8dc
Fix LibvirtGenericVIFDriver.get_config() for quota

When Nova is used with Neutron and Open vSwitch, network quotas on the
bandwidth are ignored by libvirt: the "<bandwidth>" tag is not generated in
libvirt.xml of the virtual machine. This change fixes this issue but also for
the network types: "bridge", "network" and "direct".

Commands to create the flavor with download limited to 1 MB/sec and create a
virtual machine with this flavor:
---
nova flavor-create --ephemeral=1 victor_test_vif 50 256 1 1
nova flavor-key victor_test_vif set quota:vif_inbound_average=1000
nova boot --flavor=victor_test_vif --image=cirros-0.3.1-x86_64-uec victor_test
---

Commands to upload a file to this virtual machine with SCP to check that the
bandwidth is limited to 1 MB/sec (replace 10.0.0.3 with the IP of the virtual
machine):
---
ip netns
sudo ip netns exec qrouter-128db593-a0db-40c3-84c7-e6383d40c75f bash

dd if=/dev/urandom of=random10MB bs=1024 count=10240
scp random10M cirros@10.0.3:
---

Change-Id: I0f0111fc79fe90900e38df022034b208f1129088
Closes-Bug: #1254664
DB migration 209: Clean up child rows as well

Fixes bug 1252891

In order to add missing foreign key constraints, rows that reference a
nonexistant key need to be purged. Unfortunately, DB migration 209 does
not clean up any rows in child tables that themselves have a foreign key
constraint on the tables with the missing foreign keys. This can result
in an integrity violation causing the migration to fail.

When purging the rows, make sure to purge rows in any child tables first.

Change-Id: I25bd899383c72cb0670819b030759630ac89acaa
Update Instance from database after destroy

In cases where Nova falls back to simply locally deleting an
instance, the Instance object was not updated with the deleted_at
generated by the database implementation. Due to this, the
notification emitted to signify that the delete finished was
getting emitted without a deleted_at value.

This change uses the result from the instance_destroy call to
update the Instance object with the latest state from the database.
Thus, the compute.instance.delete.end notification is emitted
with the proper deleted_at value.

Closes-Bug: #1250206
Change-Id: I55ed37cf89dd6eb8ea901c6f813191a0ce1058e8
LXC: Image device should be reset in mount() and teardown()

When fixing bug#1208387, image device name is returned in function
setup_container. But it's missed to reset in function mount() and
teardown().

This patch is to fix it. We now just return the device name when
mounting disk image during setting up container and remove the
member variable 'device' from Class _DiskImage added in last patch.

Closes-bug: #1243108
Related-bug: #1208387

Change-Id: I2aaed6f3367edce4fb23b78f0e979440954609a8
Add extra_resources field to compute_nodes table

The compute_nodes table has informtion about allocation and
availability of a predefined set of resources (e.g. memory,
disk, vcpu). The extra_resources field will contain a json
string of arbirtrary information describing an extensible set
of resources that will be tracked by resource tracker plug-ins.

This differs from other fields containing json strings that have
been added recently, including pci_stats and metrics. In particular,
pci_stats is used for information about pci devices, whereas metrics
contains continuously monitored usage data and is updated at different
times to extra_resources and according to a different trigger.
Accordingly, it makes sense to seperate these into different fields.

This is part of: blueprint extensible-resource-tracking

Change-Id: I75e339c2a2827875078c937e57c37e85d3ad1852
Setup destination disk from virt_disk_size

When running live-migration --block-migrate on a qcow2 backed
VM without cow image, the destination qcow2 file should be created
with the virtual disk size. For raw images, the virt_disk_size
is set to disk_size to ensure that virt_disk_size is always the
size of the disk that should be re-created.

Update unit tests to be more strict and check for sizes to be correct.

Closes-Bug: #1257355

Change-Id: Ie3be46024f06b9f59af92f5e3918a1958386d4f1
libvirt: consider minimal I/O size when selecting cache type

The "none" cache type is not usable when a file-system
has stricter alignment and minimal I/O size requirements,
for example the xfs file-system created with 4kiB sector size,
requires all DIRECT I/O operation has to be done with
 integer multiple of 4 kiB count/size.

qemu assumes direct I/O is doable with 512 byte minimal size,
so the  _supports_direct_io method needs to check is
the direct I/O doable with 512 byte size.

Change-Id: I5e1ba4e8afa87519cb0ed1120c213f7c318ccef2
Make it possible to override test timeout value

Currently, we are setting the test timeout value globally
in .testr.conf. It could be useful to be able to override
it per test case (e.g. for migrations tests which usually
take much longer to run than any other tests).

Related-Bug: #1216851

Change-Id: Icee09609f76f4439f6d981641be521d8d695a77e
Get test_openAuth_can_refuse_None_uri to cleanup after itself

test_openAuth_can_refuse_None_uri sets the global
libvirt.allow_default_uri_connection to False but it leaves
it in this state which could cause test_tpool_execute_calls_libvirt
to fail if it ran immediately afterwards. This patch adds a cleanup
so _reset is called at the end of the test and not just if
another test in the same class is run after it.

Change-Id: I9762fcb81b2318beeb056c67bd9172056e1449ab
Closes-Bug: 1258553
Give migrations tests more time to run

Migrations tests usually take longer to run that most other
tests. Increase the time out value just for migrations tests,
so that we don't hit various time out errors running unit tests
when CI test nodes slow down a bit.

Closes-Bug: #1216851

Change-Id: I48d7c125397cd282d52eb8e1a1ccb699188d8f14
libvirt: Custom disk_bus setting is being lost on hard_reboot

Currently, a non-default disk_bus setting of and image will not be used,
when powering on an instance (or _hard_reboot).

hard_reboot() should use image_meta, if it exist, when constructing
the disk_info

Closes-Bug: #1255449

Change-Id: I477f69c47515ad22165421b4d23ac865c7869873
Log exception in _heal_instance_info_cache

Previously, if errors occurred in _heal_instance_info_cache() they would
go unnoticed as their log level was debug and the stace trace was not
displayed. This patch changes that so the exception is logged so periodic
failures here can be more easily detected and fixed.

Change-Id: I15bb3e7e29462e930f6159be9f4bbf4980b1270c
Closes-bug: #1257940
VMware: fix bug when more than one datacenter exists

In the case that there was more than one datacenter defined on the VC,
then spawning an instance would result in an exception. The reason for this
was that the nova compute would not set the correct datacenter for the
selected datastore.

The fix also takes care of the correct folder selection. This too was a
result of not selecting the correct folder for the data center.

The 'fake' configuration was updated to contain an additional data
center with its on datastore.

Closes-Bug: #1180044
Closes-Bug: #1214850

Co-authored-by: Shawn Harsock <hartsocks@vmware.com>

Change-Id: Ib61811fffcbc80385efc3166c9e366fdaa6432bd
Rename instance_type to flavor in test_utils and nova.tests.utils

Renames occurrences of 'instance_type' to 'flavor' in
nova.tests.test_utils and nova.tests.utils along with any impacted tests
using the changed parts.

Part of blueprint flavor-instance-type-dedup

Change-Id: Ie816356c176c2be61b4a346a4d47d8f3461ae84a