Fix fake_update in test_update_missing_server

In test_update_missing_server, fake_update should be used as update()
method but it is used as create() instead. In addition, the test comment
also is wrong.
This patch fixes them.

Change-Id: Ia4704413dd7355010546dba7096ed34225bb4b0f
add log exception hints in some modules

Add hints for log exception messages

Change-Id: I51b30d58bcb4b9fa8e51b29af1cfff33c0b432aa
VMware: handle case when VM snapshot delete fails

Minesweeper fails on occasion with concurrent access to files.

If a snapshot deletion fails with the backend exception
TaskInProgress then we will retry.

A generic decorator has been added to do the retry operations.

Closes-bug: #1310817

Change-Id: I8ed7a24ccd34aeea49352ac98f34ec2960edbf97
More informative nova-scheduler log after NoValidHost is caught.

This patch adds a warning log entry for when a NoValidHost
exception causes a scheduling error. The NoValidHost
exception's message will be logged before reporting that
the machine went to error. Previously, all other exceptions
would show a stack trace. However, since the NoValidHost case
is skipped, a user may not understand why his/her machine
went to an ERROR state. This patch seeks to remove any
confusion.

Change-Id: I0f01b1095251f5c659fe010c7376f3f6bc56babe
Closes-Bug: 1336977
Backport some v3 certificate API unittest to v2 API

This patch add some v3 certificate API unittest into V2 API to improve
V2 API unittesting.

Change-Id: Icf58abbf1e83636047c1f98ff80a38077e2d694b
libvirt: skip disk resize when resize_instance is False

The method finish_migration is called with an extra
parameter 'resize_instance'. It should be used to know
if it is necessary or not to resize the disks and so
call the method 'disk_resize'.

Change-Id: Ia5a9ad7994fa9abab7cf47d36468c46d97065991
Closes-Bug: #1298981
Split EC2 ID validator to validator per resource type.

Now there is only one method to validate EC2 id: validate_ec2_id().
It's used to validate both the Instance ID as well as Volume ID for
valid EC2 ID format. However exception class raised in both cases
is "InvalidInstanceIDMalformed" and EC2 error code is
InvalidInstanceID.Malformed.

This patch addresses this problem by adding two new methods:
* validate_instance_id()
* validate_volume_id()
These methods use validate_ec2_id() and raise appropriate
exception as well as appropriate EC2 error code based on
AWS EC2 Error Code documentation:
http://docs.aws.amazon.com/AWSEC2/latest/APIReference/api-error-codes.html

Change-Id: I55591b8b643bb316e5001e645d71c13094106e96
Closes-Bug: #1199308
Fix 202 responses to contain valid content.

Using a webob.exc to generate 202s makes the response return plain
text in the body, which breaks things that expect to be able to
parse data according to the content type. Using a webob.Response
object makes sure the body of the 202 response is empty.

Change-Id: Ifca3081f573407808a2196e71ac1db87180db6ec
Resolves-bug: #1356058
Fix Trusted Filter to work with Mt. Wilson `vtime`

The Trusted Filter expects that `vtime` is returned from the
attestation server in an ISO8601 format, but the Mt. Wilson
attestation server can return it in a locale appropriate string format
instead. There's no way to configure Mt.  Wilson to do the right thing
so we should just try to parse the returned `vtime` as a string
formatted date.

Closes-Bug: #1353029

Change-Id: Ic7351e0463c014321bdb4fcfeba90ac51460b325
Add unit tests for libvirt domain creation

Tests for:
nova.virt.libvirt.driver.LibvirtDriver.get_guest_config
nova.virt.libvirt.driver.LibvirtDriver._create_domain

Partially implements: bp libvirt-lxc-user-namespaces

Change-Id: I4aada9ac06cd423b04a4c6136165c65e64e378aa
Use flavor in confirm-resize to drop claim

Current the confirm_resize() clean the stashed instance_type information
before invoking the drop_resize_claim, and the drop_resize_claim() try to
fetch the old instance_type in system metadata, which will be wrong.

We change the _cleanup_stored_instance_types() to pass back the
to_be_dropped instance_type, and use that information for drop_resize_claim.

Change-Id: Iba36acc75d3840104746596977b33bcd4e010f2e
filter: add per-aggregate filter to configure max_io_ops_per_host

Adds a filter AggregateIoOpsFilter whichprovides the ability
to read from aggregates metadata "max_io_ops_per_host".

DocImpact
Implements: blueprint per-aggregate-max-io-ops-per-host
Change-Id: I086033e7904c16995298bee7196ad3e7b5dc7aaf
Raise Not Implemented error from V2 diagnostics API

In case any driver does not implement functionality of providing VM
diagnostics, HTTPNotImplemented error will be raised from driver.
Nova V2 API should catch the same and pass it with proper error message
for better debugging.

This patch catch & raise  HTTPNotImplemented exception for
V2 "os-server-diagnostics" API.

Adding UT also for the same.

Change-Id: I485d3487d01eba07e3c34f6ef9b5c707af54ebb7
Raise HTTPNotFound error from V2 cert show API

There might be possibility of not having the CA file while
requesting certificate show API. API should raise HTTPNotFound
for clear error.
This patch raise this exception for V2 API. This has been fixed in
V3 API.

Adding UT also for the same.

Change-Id: Iafb33fce6433a31f9767067bbc45b95acd4ee511
VMware: Remove references to ebs_root from spawn()

spawn() contained an important boolean called ebs_root. This was True
for any spawn which contained a block device mapping. This no longer
has any meaning, and was confusing. This change removes references to
ebs_root, instead using block_device_info directly.

N.B. The block device mapping behaviour of the VMware driver is still
not correct, but fixing it is not the purpose of this patch. This
patch simply makes the existing behaviour easier to see.

N.B. test_spawn_mask_block_device_info_password in test_vmops.py
assumes internal details of spawn in order to cause a failure after
emitting a log message. In moving the log message we need to update
the dance this test does to ensure that spawn fails after the log
message, but not before.

This change has been split out of
https://review.openstack.org/#/c/87002/, which was written by Shawn
Hartsock.

partial blueprint vmware-spawn-refactor

Co-authored-by: Shawn Hartsock <hartsocks@acm.org>
Change-Id: I11b355a6b53194bc9f85f542b54c9fbb9c061e04
VMware: Create fake VM with given datastore

FakeVim._create_vm() was creating a VM with a default datastore rather
than the given one. This prevented testing that the correct datastore
has been used.

Change-Id: I640a21f23b2d5c1186772c8c5ce3ca8927e6cbc5
VMware: Fix type of VM's config.hardware.device in fake

config.hardware.device was handled inconsistently, and could be of
various types. When created by _create_vm(), it was actually a
VirtualDeviceConfigSpec[]. It should be a VirtualDevice[].

Fix tests which depended on the wrong type.

Add _create_array_of_type function to fake to create an array type
which works correctly when passed to code. A returned array of Foo
must be an instance of class ArrayOfFoo, and contain a single
property, Foo, which is a python array of objects of type Foo.

Change-Id: Ic90e1d0d947941621606fded9e56f9a98d111cad
VMware: refactor spawn() code to build a new VM

Move the spawn() code which builds a new VM into
vmops.build_virtual_machine().

We remove the duplicate check of instance_name. Other than that this
is straight code motion.

In addition, we add a basic test of the function of
build_virtual_machine, which tests that all given configuration is
reflected in the resulting virtual machine.

This change has been split out of
https://review.openstack.org/#/c/87002/, which was written by Shawn
Hartsock.

partial blueprint vmware-spawn-refactor

Co-authored-by: Shawn Hartsock <hartsocks@acm.org>
Change-Id: Id9c95966eefb644fdaf36465e7b3d3583826314a
VMware: spawn refactor _configure_config_drive

Factor out the code to configure the virtual machine with a new
config drive iso image.

partial blueprint vmware-spawn-refactor
Change-Id: Ie387445ae83e89fc436243a4c8ee9030d5221510
Image caching tests: use list comprehension

Use a list comprehension for creating a list of instance objects
for the tests.

Change-Id: If25d024eeeb1503d7a7dc2dbc895b2bf3152c7db