libvirt: virt rescue method now supports objects

Commit a37a96fd13b933123e80ad24e37fc37d73c16059 ensured
that the compute API and manager are now getting objects
for the rescue method.

This patch adds support to the virt driver code.

Change-Id: I2ae89f937b1983a2b0599e2eded343aa8358dbba
libvirt: virt unrescue method now supports objects

Commit 65ad230af5ece020822fa963dd30fd641a54ae91 ensured
that the compute API and manager are now getting objects
for the unrescue method.

This patch adds support to the virt driver code.

Change-Id: I5cca3fa997042b164c21fe2a13c2311f4b18ec06
Fix error in rescue rpcapi that prevents sending objects

The change restores the rescue_instance rpc's ability to send
instance objects from version 3.11 onwards, that was inadvertently
broken by https://review.openstack.org/#/c/62314

Unit test modified to check that rpc version passes instance object
instead of dict.

Change-Id: Icffea82da623d3aab4661554b33bd5959bfe2bd8
Closes-Bug: #1327145
Fix run_instance() rpc method to pass instance object

Change I8a206f63cab1e4f939d2c97c258312a8fa540aff converted run_instance()
to use instance objects, but although it changed the rpc version to 3.27
it didn't update the rpcapi method, so at the moment it still passes a
dict.

Related to blueprint compute-manager-objects-juno

Change-Id: I0e511a5c4f1adce901f8812a935b98886c216dc9
Enforce task_state is None in ec2 create_image stop instance wait loop

We're hitting races in the gate where the instance.vm_state is STOPPED
but the task_state is POWERING_OFF so when the compute_api.start method
is called, we're in an invalid task state and fail.

The compute manager's stop_instance method is correctly setting the
vm_state to STOPPED and the task_state to None when the instance is
powered off via the virt driver, so we must be hitting this from races
in the ec2 API as noted in the TODO above the method definition.

This change simply checks the task_state in addition to the vm_state
in the wait loop before continuing. The error message is also updated
for context by including the instance uuid, vm_state and task_state,
and removes the timeout value in the message since it was in
milliseconds, not seconds, to begin with.

There is already a unit test that covers this change (which was racing,
hence the bug). There are no changes to that unit test since it's really
an integration test that's running through the compute API and compute
manager code, so the fix tests itself.

Closes-Bug: #1334345

Change-Id: I13f0c743cadda6439ae15607a9ef6e4e4985626d
Fix wrong lock name for operating instance external events

This patch correct the wrong lock name for operating instance events.
Currently it use function object InstanceEvents._lock_name as the lock
name, the lock name will be like "<function _lock_name at 0x7fe6a7edec08>".
So when process the server external event, it will use one lock for all
the events that may belong to different instances. This isn't efficient.
This patch correct it to invoke InstanceEvents._lock_name to get the lock
name as '<instance-uuid>-events'. Then the lock is per instance.

Change-Id: I57712afe893933fde6ea7ab3f5d6328f04c65a8a
Closes-Bug: #1330431
Pass instance to _set_instance_error_state

This builds on commit 77a8478 so that _error_out_instance_on_exception
can pass the instance to _set_instance_error_state.  The next change in
the series will pass the instance down to _instance_update so if we have
an instance object we can avoid the direct call to conductor for state
changes and just update the instance object directly, which allows us to
remove instance.refresh() in the tests to pick up out-of-band database
updates on instance objects.

Part of blueprint compute-manager-objects-juno

Change-Id: I0a4607d368d3c226ca5a60d017d27c199c0dfe89
Add __repr__ handler for NovaObjects

This adds a __repr__ implementation for the base NovaObject which
should help debugging by compactly describing the contents of an
object. This adds stringification helpers to the Field object(s),
which lets us control what each type looks in compact format. Since
this is just for debugging, we avoid recursing too deeply on nested
objects and the like.

Change-Id: Ic604acfda7e7c7adb913479355726e16ad375f1d
Sync process utils from olso

This change syncs in:

33afb20 Fix broken formatting of processutils.execute log statement
5e3d3a5 Mask passwords included without quotes at the ends of commands
6751b30 Remove extra whitespace

The most important part is '33afb20' which was broken in nova after
the last oslo sync.

Change-Id: I669e283992347328d602b5a330fde9d1981d22f2
Move the instance_type_id judgment to the except-block

If the instance types are not stashed, will fallback to the
conductor API to retrieve them. So the instance_type_id is
only used in except-block.

Change-Id: If45eb01dd8292e87eda6516cd79c94fb7f6005fb
virt: add helper module for determining VCPU topology

Introduce a nova/virt/hardware.py module which will container
helper APIs related to instance hardware configuration, for
use by virt driver impls. The first APIs added provide a
way for the virt driver to determine what VCPU topology to
expose to guest instances based on the image metadata and
flavor extra specs.

Blueprint: virt-driver-vcpu-topology
Change-Id: I419cd70497c3a4c3f18e77d75cb7e09c0b724465
Avoid referencing stale instance/network_info dicts in firewall

This makes the virt.firewall code cleaner in terms of referencing
the cached instance and network_info code it stores. Before this
patch, concurrent instance operations could modify these two dicts
so that while we're iterating instances, the network_info dict
is suddenly missing information we need.

The right fix for this is to use instance objects and their
associated info_cache objects, but that's a larger fix and one
not as well-suited to backporting to previous releases which
suffer from this as well.

The approach taken here is that we store the instance and
network_info cache together in the same dict that we can pop()
from atomically (this is not really necessary, but helps to
prevent introducing more of these cases). When we iterate over
the contents, we iterate over a copy of the keys, being careful
not to let a suddenly-missing key break us, and passing the
details all the way down the stack instead of having deeper calls
hit the cache dicts again.

Change-Id: I33366f50024a82451842d045b830ab19b59879c3
Closes-bug: #1182131
Merge "Fix error in rescue rpcapi that prevents sending objects"
Merge "Add __repr__ handler for NovaObjects"
Merge "Move the instance_type_id judgment to the except-block"
Merge "Avoid referencing stale instance/network_info dicts in firewall"
Merge "Pass instance to _set_instance_error_state"
Merge "Sync process utils from olso"
Merge "Fix wrong lock name for operating instance external events"
Merge "Enforce task_state is None in ec2 create_image stop instance wait loop"