Process power state syncs asynchronously

The instance locking introduced in
cc5388bbe81aba635fb757e202d860aeed98f3e8 keeps the power state sane
between stop and the periodic task power sync.  However, locking on
an instance in the periodic task thread can potentially lock
that thread for a long time.

Fix this condition by executing power syncs in a background GreenPool.
Limit the syncs to 1 at a time per instance.

Closes-Bug: 1363231

Change-Id: Idbc00ddfcb7a6816433863ed1686ad8f7477d99b
Add new unit tests for PCI stats

These unit tests make sure that the stats pools are created and
ordered properly.

Partially Implements: blueprint pci-passthrough-sriov

Change-Id: I81591b18d1045a4451ce009b091b22875eb04890
Stash numa-related flavor extra_spec items in system_metadata

This is a temporary measure to avoid having to add additional persistence
of extra_specs data simply to facilitate merging the NUMA patches. Only
the numa_cpus.* and numa_mem.* items are saved from extra_specs at
this time. The expectation is that this will quickly come out in kilo
when the rest of the flavor information stashed in system_metadata gets
removed and stored elsewhere.

Blueprint: virt-driver-numa-placement
Change-Id: I83ca863d521c579d8f1fe6168d7b1096af9f14f1
Hook NUMA topology checking into claims

This patch adds the ability to test claims against NUMA topology exposed
by the host and instance. This is currently a no-op as no code in the
scheduler will actually attempt to set the limits against numa_topology.
Since getting the NUMA topology of the instance may require an additional
db roundtrip, we try to avoid getting it unless absolutely necessary.

This patch also changes the resource_tracker resize_claim method to
accept the image_metada of the instance as it is used (alongside flavor)
to calculate the new topology that will be exposed after the resize is
done.

The resource tracker will not do any actual usage tracking yet - this
will be done in a follow-up patch.

Blueprint: virt-driver-numa-placement
Change-Id: I07b690243d481c855c5e706f968cf224b336a1a0
Make resource tracker track NUMA usage

This patch makes the resource tracker actually track and update NUMA
usage from instance andd resize claims, and periodic resource updates.

Change-Id: Iea9d076cb37e530f442faefacd2a59b6b6225e92
Blueprint: virt-driver-numa-placement
API boot process sets NUMA topology for instances

This patch makes the API instance boot path look at the bits that can
contain requested topology, generate the desired topology and make sure
it is saved on the instance itself.

We also need to account for the flavor passed to
VirtNUMAInstanceTopology.get_constraints not being an actual Flavor
object, and work with db layer objects.

Change-Id: Ia3f3446da99a04e2d8152acb7d0bda7a1d502c83
Blueprint: virt-driver-numa-placement
Make HostManager track NUMA usage

This patch adds tracking of NUMA usage in scheduler's HostManager class,
by adding it to the HostState, and making sure it gets updated along
with all the other usage data.

Change-Id: Id31c9a7fd54b918b3d38abb61b305cf7048a6a7d
Blueprint: virt-driver-numa-placement
Add the NUMATopologyFilter

This patch adds the basic NUMA filter that will take the proposed
instance topology and try to match it against that of a host.

The matching will be done in the following manner:

 * Filter will try to match the exact NUMA cells of the instance to those
   of the host. It *will not* attempt to pack the instance onto the host.
 * It will consider the standard over-subscription limits for each cell,
   and provide limits to the compute host accordingly.
 * If instance has no topology defined, it will be considered for any host.
 * If instance has a topology defined, it will be considered only for NUMA
   capable hosts.

DocImpact

Change-Id: I8788dde69524c8a32a41ce31a96c89f9b09e91ce
Blueprint: virt-driver-numa-placement
libvirt: Consider numa_topology when booting

This patch hooks all the already introduced pieces together to enable
NUMA aware placement of instances.

The libvirt driver boot process now looks into the NUMA topology field
of both instance and the host it is being booted on, and based on that
adds some more configuration to the generated guest XML.

In case the host is NUMA capable, but the instance has not requested
a NUMA topology, we will try to pack the instance into a single cell.
If that fails, currently we will not make further attempts (although
an improvement could be to expose the underlying topology to the
instance).

In case the host is NUMA capable, and instance has requested a specific
NUMA topology, we will try to pin the vCPUs of different instance NUMA
cells to the corresponding host NUMA cells, and also expose the instance
NUMA topology to the guest OS.

We will still try to respect CONF.vcpu_pin_set option and avoid pinning
to CPUs that are not in this set.

DocImpact

Change-Id: I611d44ab75475028f63c2b7f8a0d2ef37d85be6e
Blueprint: virt-driver-numa-placement
Merge "Add new unit tests for PCI stats"
Merge "Process power state syncs asynchronously"
Merge "Stash numa-related flavor extra_spec items in system_metadata"
Merge "Hook NUMA topology checking into claims"
Merge "Make resource tracker track NUMA usage"
Merge "API boot process sets NUMA topology for instances"
Merge "Make HostManager track NUMA usage"
Merge "Add the NUMATopologyFilter"
Merge "libvirt: Consider numa_topology when booting"