Fix config generator to use keystonemiddleware

The auth_token middleware was moved from keystoneclient.middleware
to keystonemiddleware. The config generator is configured to point
to the new location.

Closes-Bug: #1366594
Change-Id: I5543973b2c7ac06c048686c24354ecf6eb251027
Ironic:  Fix direct use of flavor and instance module objects

This replaces all uses of nova.objects.flavor_obj.Flavor* with
nova.objects.Flavor* and nova.objects.instance_obj.Instance* with
nova.objects.Instance*

This work was done throughout Nova. An example of this is
commit 3c1085f528596089a638ac3bb44a9ed3e201e62f.

Change-Id: I8f2dbb408acbdf8124bc141925de03ce6f12bd91
Add InstancePCIRequests object

Part of blueprint pci-passthrough-sriov

Co-Authored-By: Dan Smith <dansmith@redhat.com>
Co-Authored-By: Baodong (Robert) Li <baoli@cisco.com>
Change-Id: I125e4afdf98a7073abcb8117bbb7a3e3fe2f9830
Handle exception better in v2 attach_interface

Add more information when exception occurs in attach interface
at v2 API layer.

Change-Id: I7402cfc28d4ee7989c7eb054556df57918bbaeee
Stop stack tracing when trying to auto-stop a stopped instance

Commit cc5388bbe81aba635fb757e202d860aeed98f3e8 added locks to
stop_instance and the _sync_power_states periodic task to try and fix a
race between stopping the instance via the API where the task_state is
set to powering-off, and the periodic task seeing the instance
power_state as shutdown in _sync_instance_power_state and calling the
stop API again, at which point the task_state is already None from the
first stop API call and we get an UnexpectedTaskStateError.

The handle_lifecycle_event method is getting callbacks from the libvirt
driver on state changes on the VM and calling the
_sync_instance_power_state method which may try to stop the instance
asynchronously, and lead to UnexpectedTaskStateError if the instance is
already stopped by the time it gets the lock and the task_state has
changed.

Attempting to lock in handle_lifecycle_event just moves the race around
so this change adds logic to stop_instance such that if the instance
says it's active but the virt driver says it's not running, then we add
None to the expected_task_state so we don't stacktrace on
instance.save().

An alternative and/or additional change to this would be doing a call
rather than a cast when _sync_instance_power_state calls the stop API
but in some previous testing it doesn't appear to make a significant
difference in the race found when we hit the stop_instance method.

Adds a bunch of debug logging since this code is inherently racey and
is needed when looking at failures around these operations.

Closes-Bug: #1339235
Closes-Bug: #1266611
Related-Bug: #1320628

Change-Id: Ib495a5ab15de88051c5fa7abfb58a5445691dcad
Replace pci_request flavor storage with proper object usage

This makes the PCI code use the InstancePCIRequests object to store
and retrieve the pci_request information, instead of stashing it in
system_metadata.

Related to blueprint pci-passthrough-sriov

Co-Authored-By: Baodong (Robert) Li <baoli@cisco.com>
Change-Id: I012ee5c118265e044ff41fb58b732728946ee85a
Add request_id in PciDevice

Added request_id in the PciDevice object and the corresponding
model. Migration script is added accordingly. Parent objects's
versions are bumped, and backleveling rules added.

Partially Implements: blueprint pci-passthrough-sriov
Change-Id: I2367339a29c0da8b526fd5105766e51ca4e8c6ab
Support SR-IOV networking in the PCI modules

This patch takes care of changing the PCI modules in support of SR-IOV
networking:
  -- Updated pci_whitelist to use devspec that is introduced in this
     patch: https://review.openstack.org/#/c/99043/
  -- For stats pool, to keep the existing behavior, pool keys always
     include product_id and vendor_id. Meanwhile, to work with tags
     and support pci requests based on tags, pools are ordered based
     on the number of keys of the pool. This makes sure that legacy
     pci requests will be satisfied with the pools in the beginning,
     possiblly with devices that are not tagged.

Partially Implements: blueprint pci-passthrough-sriov
Change-Id: I9da7bd921f098c958d6fb711a4d03a76862e95c7
Support SR-IOV networking in nova compute api and nova neutronv2

Add another field into the NetworkRequest object for the request_id.
For a SR-IOV port, Create a PCI request and add the pci-request-id into
the tuple. For a non SR-IOV port, the pci-request-id is None.
Add new fields 'profile' and 'vnic_type' in VIF.
Enhance nova neutronv2 api module to:
  -- retrieve a port's vnic-type and the physical network it is attached to
  -- populate binding:profile before updating a neutron port
  -- populate VIF with newly added fields.
Partially Implements: blueprint pci-passthrough-sriov
Change-Id: I4aab1cd43960d5b1b54a67a05c191f295dfc46fb
Support SR-IOV networking in libvirt

This patch takes care of changing the libvirt modules in support of SR-IOV
networking. Added full support in configuring the two VIF types:
VIF_TYPE_802QBH and VIF_TYPE_HW_VEB and in generating the corresponding
domain XML. It also adds support of attaching/detaching SR-IOV ports
on instance suspend/snapshot/resume operations.

Change-Id: Ibc47d13f47fa0b41cefc27bfff942cdbba6eb399
Implements: blueprint pci-passthrough-sriov
Adds LOG statements in multiprocess API test

Renames _terminate_workers() to _wait_for_workers_to_end() to accurately
match what is done in that method. Adds a bunch of LOG.info() and
LOG.warn() calls in the various methods in the test to get better
debugging information when bug # 1357578 rears its ugly head.

Change-Id: I6698b7c71ec651c812138c8dd93b1e1b33ee1178
Related-bug: #1357578
Clarify logging in lockutils

The old logging statements in lockutils made for some confusing log
entries where it would appear a given lock was acquired multiple
times by different threads at the same time.  See referenced bug
for details.

In order to alleviate that confusion, this change does a few things:
1) Adds an explicit "acquired" message inside the lock so it is
   clear when the lock was actually acquired.
2) Moves the release message inside the semaphore so there's no
   chance of it being logged out of order.
3) Removes the "Got semaphore" message and splits it into two
   separate messages depending on whether the semaphore was found
   in the weakref dictionary.  Making it clear which code path
   was followed should help with future debugging.

This is the Nova port of oslo-incubator change
If863c9d3896bc786f3c9c46196ebb0766e57d863

Change-Id: I530befb42a963a40e7d5bf8d4d81fc097e1cb309
Partial-Bug: #1367941
Merge "Stop stack tracing when trying to auto-stop a stopped instance"
Merge "Ironic:  Fix direct use of flavor and instance module objects"
Stop using intersphinx

Remove intersphinx from the docs build as it triggers network calls that
occasionally fail, and we don't really use intersphinx (links other
sphinx documents out on the internet)

This also removes the requirement for internet access during docs build.

This causes docs jobs to fail because we error out on warnings.

Change-Id: I71e941e2a639641a662a163c682eb86d51de42fb
Related-Bug: #1368910
Merge "Fix config generator to use keystonemiddleware"
Merge "Adds LOG statements in multiprocess API test"
Merge "Handle exception better in v2 attach_interface"
Merge "Add InstancePCIRequests object"
Merge "Replace pci_request flavor storage with proper object usage"