Make separate calls to libvirt volume

This is a follow up patch to
https://review.openstack.org/#/c/116998/, where libvirt
volume's connect_volume method were split into get_config and
connect_volume. The following patch changes libvirt driver
to call the appropriate volume method, get_config or
connect_volume, instead of the old monolithic call to
connect_volume. connect_volume will no longer need to return
the libvirt xml after this switch over. This will improve
performance and eliminate possible race conditions.

Change-Id: I0d0d7ba98c53ad7509c28bb4866c488c0a049d2f
Closes-Bug: #1359617
Remove outdated docstring for XenApi driver's options

XenAPI's driver options's help strings can explain how to
use these options, so remove these duplicated and outdated
docstring.

Change-Id: I3de257e75ba13352e9460070f5bb2a6d7cf36a4e
Clean up iSCSI multipath devices in Post Live Migration

When a volume is attached to a VM in the source compute node through
multipath, the related files in /dev/disk/by-path/ are like this

stack@ubuntu-server12:~/devstack$ ls /dev/disk/by-path/*24
/dev/disk/by-path/ip-192.168.3.50:3260-iscsi-iqn.1992-04.com.emc:cx.
fnm00124500890.a5-lun-24
/dev/disk/by-path/ip-192.168.4.51:3260-iscsi-iqn.1992-04.com.emc:cx.
fnm00124500890.b4-lun-24

The information on its corresponding multipath device is like this
stack@ubuntu-server12:~/devstack$ sudo multipath -l 3600601602ba034
00921130967724e411
3600601602ba03400921130967724e411 dm-3 DGC,VRAID
size=1.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=-1 status=active
| `- 19:0:0:24 sdl 8:176 active undef running
`-+- policy='round-robin 0' prio=-1 status=enabled
  `- 18:0:0:24 sdj 8:144 active undef running

But when the VM is migrated to the destination, the related information is
like the following example since we CANNOT guarantee that all nodes are able
to access the same iSCSI portals and the same target LUN number. And the
information is used to overwrite connection_info in the DB before the post
live migration logic is executed.

stack@ubuntu-server13:~/devstack$ ls /dev/disk/by-path/*24
/dev/disk/by-path/ip-192.168.3.51:3260-iscsi-iqn.1992-04.com.emc:cx.
fnm00124500890.b5-lun-100
/dev/disk/by-path/ip-192.168.4.51:3260-iscsi-iqn.1992-04.com.emc:cx.
fnm00124500890.b4-lun-100

stack@ubuntu-server13:~/devstack$ sudo multipath -l 3600601602ba034
00921130967724e411
3600601602ba03400921130967724e411 dm-3 DGC,VRAID
size=1.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=-1 status=active
| `- 19:0:0:100 sdf 8:176 active undef running
`-+- policy='round-robin 0' prio=-1 status=enabled
  `- 18:0:0:100 sdg 8:144 active undef running

As a result, if post live migration in source side uses <IP>, <IQN> and
<TARGET LUN Number> to find the devices to clean up, it may use 192.168.3.51,
iqn.1992-04.com.emc:cx.fnm00124500890.a5 and 100.
However, the correct one should be 192.168.3.50, iqn.1992-04.com.emc:cx.
fnm00124500890.a5 and 24.

Similar philosophy in (https://bugs.launchpad.net/nova/+bug/1327497) can be
used to fix it: Leverage the unchanged multipath_id to find correct devices
to delete.

Change-Id: I875293c3ade9423caa2b8afe9eca25a74606d262
Closes-Bug: #1357368
VMware: add tests for spawn with config drive enabled

Add tests for test_vmops where config driver is enabled.

TrivialFix

Change-Id: I6ca9bfb32672d42987892ee4e95c0d33c3ef2685
VMware: fix compute node exception when no hosts in cluster

Commit 4033c0c9c16d2844b36fd5627717e7ce206887f6 casues the
regression. The result was a empty string and not None.
That is suds returns a Text object and not None.

Change-Id: I79e70e300f40eb3561cf56478d578f0f8cda273e
Closes-bug: #1383305
Use wsgi.response for v2.1 API

Now a decorator @wsgi.response() can be used for specifying success
status code. Most v2.1 APIs use the decorator and it makes the code
more readable. This patch replaces webob.Response with the decorator
for the other APIs.
In addition, this patch changes some orders of decorators for showing
success code and expected error codes clearly.

Even if applying this patch, some webob.Response() calls still exist
under the path nova/api/openstack/compute/plugins/v3/ because these
cases need to contain some information in a response header.

Partially implements blueprint v2-on-v3-api

Change-Id: I419819b8e55427979e8d82b972cd237952a5b06b
VMware: Add fake.create_vm()

Add a utility function for creating VMs in tests. Based on the
existing FakeVim._create_vm function which implements CreateVM_Task.
FakeVim._create_vm is refactored to call create_vm.

Change-Id: I4949408420704cd7e6dd4fc11226472362422a93
update retryable errors & instance fault on retry

HTTP errors can be split into a few categories: client ephemeral, server
ephemeral, server permanent, and globally permanent. You could argue
there is even more permutations. However, for simplicity, these errors
can be viewed as ephemeral and permanent.

The Glance XenAPI plugin has been updated to raise a RetryableError for
ephemeral and unexpected (i.e. errors that are categorized as neither
permanent or ephemeral) errors.

Additionally, an instance fault will be logged every time an error
occurs. This will serve as transaction history where every attempt is a
transaction on the state. If an ephemeral error occurs, there is a
retry, then a permanent error, the history of each error will be in the
instance_faults.

Deployers should configure the num_retries relative to the number of
api_servers. Right now, servers are put in random order and then cycled.
Trying the same server multiple times could cause unnecessary load and
delays. However, trying multiple servers, is ideal when a single server
is behaving badly or cannot reach another server it needs to communicate
with to fulfill a request.

Closes-Bug: 1380776

Change-Id: I267a5b524c3ff8a28edf1a2285b77bb09049773c
Read flavor even if it is already deleted

Currently libvirt code has some references to Flavor
as objects.Flavor.get_by_id. because the flavor might be
deleted before it's referenced, so most code need to
read deleted flavor in libvirt driver. This patch
makes the _prepare_args_for_get_config which is used for
sriov feature consistent to other Flavor get functions.

Related-Bug: #1352668

Change-Id: Ia4a03467720f1fed18cabe1b78bdd4488f316e34
Use common get_instance call in API plugins

Cleanup API to use common get_instance call which
automatically does exception handling. Not all cases
are handled in this patch as some require more
significant testcase rework and are done in future
patches to make it easier to review.

Change-Id: Id27350e6562636f81347b490db649744b6b41186
Split and fix the type filters unit tests

The unit test for TypeAffinityFilter was broken -- it was passing in the
instance_type['name'] key, when the filter class actually checks the
'id' key of the instance_type property of the filter_properties dict. :(

This corrects the tests, removes all DB accesses from both the type and
aggregate type affinity filter unit tests, and splits them out into
their own test file.

Change-Id: I5fc27c538cc419b4ed972839751e0e779151fc58
Split out num instances filter unit tests

Break out the unit tests for num instances filters from
test_host_filters.py into its own file. In the process, we mock out the
database calls for getting aggregate information instead of creating
database records.

Change-Id: I71516cee8b1660e8e8f76eae4ae831b34232289e
Split out IO Ops filter unit tests

Break out the unit tests for IO Ops filters from test_host_filters.py
into its own file. In the process, we mock out the database calls for
getting aggregate information instead of creating database records.

Change-Id: I8395ba7c09ca5c85ecd94d2cc670f3cd0880a373
Split out core filter unit tests

Break out the unit tests for core filters from test_host_filters.py
into its own file. In the process, we mock out the database calls for
getting aggregate information instead of creating database records.

Change-Id: I9c602b9832632ad84b4b0941c5254a283d278b69
Split out aggregate disk filter unit tests

The aggregate disk filter unit tests remained in test_host_filters.py,
and this patch moves them to the test_disk_filters.py file. This was the
last occurrence of aggregate database calls from test_host_filters.py.

Change-Id: Ic47c41474dce8bf3936910be5feee081445193de
Ignore DiskNotFound error on detaching volumes

When volume_api.terminate_connection RPC calls fail due to timeout,
detaching volume will be impossible because it is already detached but
still have the volume mapping entry. To make it detachable completely
by retrying volume detach, this patch make it ignore DiskNotFound error
while detaching volumes and go through removing the mapping.

Change-Id: I95077179160a2eb35436db2a5f7a5fbca64ec323
Closes-Bug: #1367964
Remove warning & change @periodic_task behaviour

During development of Juno, Ia227f4c4e69ecf361ab02d1d17a3010303650104
added a warning that the behaviour of periodic_tasks with an interval of
0 would change.  This patch changes the behaviour and removes the
warning.

Oslo (which provides the @periodic_task decorator) uses spacing=0 to
mean "run at the default interval of 60s", but several nova methods used
spacing=0 to mean "don't run".  These methods now behave consistently
with the Oslo code.

Due to an error in the original patch, some of the warnings refer to the
intended behaviour change as being in "the K release", and some to "Juno".
The warning about changing behaviour is first present in Juno, so this
patch should be merged in Kilo.

Change-Id: Ie5f2092a3b84e3674fe6a68ce511d2dc218bb625
Closes-bug: 1276203
Delay STOPPED lifecycle event for Xen domains

When using libvirt, a reboot from inside of a kvm VM doesn't trigger
any libvirt lifecycle event. That's fine. But rebooting a Xen VM leads
to the events VIR_DOMAIN_EVENT_STOPPED and VIR_DOMAIN_EVENT_STARTED.
Nova compute manager catches these events and tries to sync the power
state of the VM with the power state in the database. In the case the VM
state is ACTIVE but the power state is something that doesn't fit, the
stop API call is executed to trigger all stop hooks. This leads to the
problem that a reboot of a Xen VM without using the API isn't possible.
To fix it, delay the emission of the STOPPED lifecycle event a couple of
seconds. If a VIR_DOMAIN_EVENT_STARTED event is received while the STOPPED
event is pending, cancel the pending STOPPED lifecycle event so the VM
can continue to run.

Closes-Bug: #1293480
Change-Id: I690d3d700ab4d057554350da143ff77d78b509c6
Merge "Remove warning & change @periodic_task behaviour"
Merge "Split and fix the type filters unit tests"