add log exception hints in some modules

Add hints for log exception messages

Change-Id: I51b30d58bcb4b9fa8e51b29af1cfff33c0b432aa
VMware: handle case when VM snapshot delete fails

Minesweeper fails on occasion with concurrent access to files.

If a snapshot deletion fails with the backend exception
TaskInProgress then we will retry.

A generic decorator has been added to do the retry operations.

Closes-bug: #1310817

Change-Id: I8ed7a24ccd34aeea49352ac98f34ec2960edbf97
Backport some v3 certificate API unittest to v2 API

This patch add some v3 certificate API unittest into V2 API to improve
V2 API unittesting.

Change-Id: Icf58abbf1e83636047c1f98ff80a38077e2d694b
filter: add per-aggregate filter to configure max_io_ops_per_host

Adds a filter AggregateIoOpsFilter whichprovides the ability
to read from aggregates metadata "max_io_ops_per_host".

DocImpact
Implements: blueprint per-aggregate-max-io-ops-per-host
Change-Id: I086033e7904c16995298bee7196ad3e7b5dc7aaf
Raise HTTPNotFound error from V2 cert show API

There might be possibility of not having the CA file while
requesting certificate show API. API should raise HTTPNotFound
for clear error.
This patch raise this exception for V2 API. This has been fixed in
V3 API.

Adding UT also for the same.

Change-Id: Iafb33fce6433a31f9767067bbc45b95acd4ee511
VMware: Create fake VM with given datastore

FakeVim._create_vm() was creating a VM with a default datastore rather
than the given one. This prevented testing that the correct datastore
has been used.

Change-Id: I640a21f23b2d5c1186772c8c5ce3ca8927e6cbc5
VMware: Fix type of VM's config.hardware.device in fake

config.hardware.device was handled inconsistently, and could be of
various types. When created by _create_vm(), it was actually a
VirtualDeviceConfigSpec[]. It should be a VirtualDevice[].

Fix tests which depended on the wrong type.

Add _create_array_of_type function to fake to create an array type
which works correctly when passed to code. A returned array of Foo
must be an instance of class ArrayOfFoo, and contain a single
property, Foo, which is a python array of objects of type Foo.

Change-Id: Ic90e1d0d947941621606fded9e56f9a98d111cad
VMware: refactor spawn() code to build a new VM

Move the spawn() code which builds a new VM into
vmops.build_virtual_machine().

We remove the duplicate check of instance_name. Other than that this
is straight code motion.

In addition, we add a basic test of the function of
build_virtual_machine, which tests that all given configuration is
reflected in the resulting virtual machine.

This change has been split out of
https://review.openstack.org/#/c/87002/, which was written by Shawn
Hartsock.

partial blueprint vmware-spawn-refactor

Co-authored-by: Shawn Hartsock <hartsocks@acm.org>
Change-Id: Id9c95966eefb644fdaf36465e7b3d3583826314a
VMware: spawn refactor _configure_config_drive

Factor out the code to configure the virtual machine with a new
config drive iso image.

partial blueprint vmware-spawn-refactor
Change-Id: Ie387445ae83e89fc436243a4c8ee9030d5221510
Share unittest between v2 and v2.1 for hide_server_addresses extension

This patch make the v2.1 hide_server_addresses extension running on v2
unittest, also delete the v3 unittest.

Partially implements blueprint v2-on-v3-api

Change-Id: I9b8e8d10ba59b344d5bdf285fffffd98ed02ebc0
Share common test settings in test_flavor_manage

In PrivateFlavorManageTest, there are duplicated test setting code in each
test. This patch moves these settings into setUp() method to reduce code.
In addition, this patch adds an internal method also for the same purpose.

Change-Id: I31883497174141ac4034922981e0a846f1fc2deb
Fix V2 unit tests to test hypervisor API as admin

Hypervisors API are Admin API and unit tests should test those accordingly.
But All the V2 hypervisors Unit tests tests those as a non Admin API

This patch fix this issue and add more non-admin unit tests

Change-Id: I5c29828d567e2044786d4bcbff4970219c6e8c5f
Closes-Bug: #1360113
Avoid refreshing PCI devices on instance.save()

Currently the way we refresh Instance.pci_devices causes current-version
PciDevice and PciDeviceList objects to be sent back to a potentially old
client. This is because remotable methods don't undergo backporting
obligatorily in the way that remotable_classmethod ones do. This is
currently a problem for pci_devices and security_groups, although it is
handled differently for things like info_cache so that it's not a
problem.

The resolution to this is rather complicated, so this patch just avoids
the implicit update (which is probably never used and actually generating
some more DB traffic) for the time being.

Change-Id: I33e584c42c9484912ccc3d38b67a6a7ee3d69a10
Change v3 admin-password to v2.1

This patch changes v3 admin-password API to v2.1 and backport v3
unit test cases to v2.1.
The differences between v2 and v3 are described on the wiki page
https://wiki.openstack.org/wiki/NovaAPIv2tov3

Partially implements blueprint v2-on-v3-api
Change-Id: I167753eddd188120236c51bade549c907c8a8466
Add numa_topology column to the compute_node table

This patch adds the migration to add the numa_topology column to the
compute_node table. It also adds the field to the ORM model and the
Nova object class.

Change-Id: I4ca994662ed700a907f78aa04016b626fc351b25
Blueprint: virt-driver-numa-placement
Add a Set and SetOfIntegers object fields

Object fields that represent sets of things. Will be useful for NUMA
topology object which will be added in subsequent patches.

Change-Id: I66d61c760a70ad661dff4c40c6933df06192ee8f
Blueprint: virt-driver-numa-placement
Change v3 extended_status to v2.1

This patch changes v3 extended_status API to v2.1 and makes v2
unit tests share between v2 and v2.1.
Revert v3 changes back to v2.
os-extended-status:* -> OS-EXT-STS:*
The differences between v2 and v3 are descibed on the wiki page
https://wiki.openstack.org/wiki/NovaAPIv2tov3.
Partially implements blueprint v2-on-v3-api

Change-Id: Ieefca1d0e8da831830f43cc83eb50bc7e7f4a10c
Merge "Backport some v3 certificate API unittest to v2 API"
Merge "Change v3 extended_status to v2.1"
Allow attaching external networks based on configurable policy

Commit da66d50010d5b1ba1d7fc9c3d59d81b6c01bb0b0 restricted
attaching external networks to admin clients. This patch changes
it to a policy based check instead with the default setting being
admin only. This allows operators to more precisely configure who
they wish to allow to attach external networks without having to
give them admin access

Change-Id: I59e71f117f889f2abffddc36c1870ef1e0fe3711
DocImpact: Adds network:attach_external_network policy
Closes-Bug: #1352102