Handle a flavor create failed better

Catch FlavorCreateFailed exceptions which when flavor creation fails.
This type of flavor creation failure is caused by an internal nova problem.
Return an internal server error (500) to the user so they know
to contact their cloud provider.

Closes-Bug: #1292309

Change-Id: I268c84dae290b0170448d789509af781555df9b0
libvirt: VM diagnostics (v3 API only)

There is no formal definition for the VM diagnostics. For the
V2 API the diagnostics will be returned as they are today.

This will support backward compatibility with the existing API's.

Part of the blueprint v3-diagnostics

DocImpact

Change-Id: I4128e76622e73a40d25fb50b10b1c1ac710a5cc4
Network: add in a new network type - DVS

The support enables DVS network backends. In this case neutron
will return a vif type of DVS.

The name of the VC port group will be of the following format:
    <network-name>-<network-uuid>

The network name is so that the prefix will be easy for people to
understand. The UUID is to ensure that it will be unique.

Change-Id: Id125797632591bd799e215da7cbae07154e90344
add log exception hints in some modules

Add hints for log exception messages

Change-Id: I51b30d58bcb4b9fa8e51b29af1cfff33c0b432aa
Optimize instance_floating_address_get_all

Removed one extra round-trip to a database
and iteration through 'fixed_ip_ids' list

Change-Id: I65febddbc48350da93179aee8ff3c4130a6671d4
VMware: handle case when VM snapshot delete fails

Minesweeper fails on occasion with concurrent access to files.

If a snapshot deletion fails with the backend exception
TaskInProgress then we will retry.

A generic decorator has been added to do the retry operations.

Closes-bug: #1310817

Change-Id: I8ed7a24ccd34aeea49352ac98f34ec2960edbf97
More informative nova-scheduler log after NoValidHost is caught.

This patch adds a warning log entry for when a NoValidHost
exception causes a scheduling error. The NoValidHost
exception's message will be logged before reporting that
the machine went to error. Previously, all other exceptions
would show a stack trace. However, since the NoValidHost case
is skipped, a user may not understand why his/her machine
went to an ERROR state. This patch seeks to remove any
confusion.

Change-Id: I0f01b1095251f5c659fe010c7376f3f6bc56babe
Closes-Bug: 1336977
Backport some v3 certificate API unittest to v2 API

This patch add some v3 certificate API unittest into V2 API to improve
V2 API unittesting.

Change-Id: Icf58abbf1e83636047c1f98ff80a38077e2d694b
libvirt: Remove unecessary suffix defaulting

The defaulting of suffix is being done in function signature, so the
additional check in the function body is unecessary.

Change-Id: I8fb6c2d0d71761818430b5d6500e49fc70dad890
filter: add per-aggregate filter to configure max_io_ops_per_host

Adds a filter AggregateIoOpsFilter whichprovides the ability
to read from aggregates metadata "max_io_ops_per_host".

DocImpact
Implements: blueprint per-aggregate-max-io-ops-per-host
Change-Id: I086033e7904c16995298bee7196ad3e7b5dc7aaf
Fix NotImplementedError in floating-ip-list

A list of existing floating IP (nova floating-ip-list) fails
when using VMwareVCDriver/LibvirtDriver and nova-network.
The following error is thrown: "NotImplementedError: Cannot load
'instance' in the base class". The problem is caused by
_normalize_ip, which tries to set an 'instance' field in
FloatingIP that is needed by _translate_floating_ip_view. The
instance is derived from the FixedIP that is assigned to the
'fixed_ip' field in FloatingIP. However, FixedIP may not have
an 'instance' field set, which can cause the NotImplementedError.

The following patch removes _normalize_ip and assigns the
'instance_id' required by _translate_floating_ip_view to
the 'instance_uuid' in FixedIP, since only the instance ID
is required and not the entire instance object.

Change-Id: I2d7937e969bd87e53557842a22f3861f0d307b5e
Closes-Bug: #1356051
Raise HTTPNotFound error from V2 cert show API

There might be possibility of not having the CA file while
requesting certificate show API. API should raise HTTPNotFound
for clear error.
This patch raise this exception for V2 API. This has been fixed in
V3 API.

Adding UT also for the same.

Change-Id: Iafb33fce6433a31f9767067bbc45b95acd4ee511
VMware: Create fake VM with given datastore

FakeVim._create_vm() was creating a VM with a default datastore rather
than the given one. This prevented testing that the correct datastore
has been used.

Change-Id: I640a21f23b2d5c1186772c8c5ce3ca8927e6cbc5
VMware: Fix type of VM's config.hardware.device in fake

config.hardware.device was handled inconsistently, and could be of
various types. When created by _create_vm(), it was actually a
VirtualDeviceConfigSpec[]. It should be a VirtualDevice[].

Fix tests which depended on the wrong type.

Add _create_array_of_type function to fake to create an array type
which works correctly when passed to code. A returned array of Foo
must be an instance of class ArrayOfFoo, and contain a single
property, Foo, which is a python array of objects of type Foo.

Change-Id: Ic90e1d0d947941621606fded9e56f9a98d111cad
VMware: refactor spawn() code to build a new VM

Move the spawn() code which builds a new VM into
vmops.build_virtual_machine().

We remove the duplicate check of instance_name. Other than that this
is straight code motion.

In addition, we add a basic test of the function of
build_virtual_machine, which tests that all given configuration is
reflected in the resulting virtual machine.

This change has been split out of
https://review.openstack.org/#/c/87002/, which was written by Shawn
Hartsock.

partial blueprint vmware-spawn-refactor

Co-authored-by: Shawn Hartsock <hartsocks@acm.org>
Change-Id: Id9c95966eefb644fdaf36465e7b3d3583826314a
VMware: spawn refactor _configure_config_drive

Factor out the code to configure the virtual machine with a new
config drive iso image.

partial blueprint vmware-spawn-refactor
Change-Id: Ie387445ae83e89fc436243a4c8ee9030d5221510
Share unittest between v2 and v2.1 for hide_server_addresses extension

This patch make the v2.1 hide_server_addresses extension running on v2
unittest, also delete the v3 unittest.

Partially implements blueprint v2-on-v3-api

Change-Id: I9b8e8d10ba59b344d5bdf285fffffd98ed02ebc0
Share common test settings in test_flavor_manage

In PrivateFlavorManageTest, there are duplicated test setting code in each
test. This patch moves these settings into setUp() method to reduce code.
In addition, this patch adds an internal method also for the same purpose.

Change-Id: I31883497174141ac4034922981e0a846f1fc2deb
objects: Fix InstanceGroup.obj_make_compatible()

The obj_make_compatible() method of the InstanceGroup object
incorrectly assumed the format of the version argument was the version
broken down into int() parts.  It comes in as a string and must be
converted.

Change-Id: Id2724c5029202f001e5a2b4371f030b65c72fdfd
objects: Add base test for obj_make_compatible()

A previous commit fixed a bug in obj_make_compatible() of the
InstanceGroup object.  While fixing that, I added a test targeted at
that object.  This patch adds a test that calls obj_make_compatible()
on every past version of every object.  It doesn't actually test that
the data is converted, but it makes sure that calling the method
doesn't blow up on something basic like expecting the wrong format of
the version parameter.

Only one object had to change for this test to work, and that was a
test object.  It assumed a field was always set, and it's not in the
case of this test.  However, the field in question is not actually set
as nullable, so it *should* be set.  However, the way these methods
are implemented elsewhere, they never assume the field to be set and
present in the primitive version of the object.

Change-Id: Id56b2f9763cfcb99a3c4a2d56329485c5e9c3b5c