Add plan for kilo blueprints: when is a blueprint needed

At the nova mid-cycle we agreed that while specs have made our
process significantly better, we think the current approach still leaves
much to be desired. This document is the outcome of the discussion at
the mid-cycle on how we want to manage blueprints in kilo.

This proposal is being placed in devref, as we want to use gerrit's
workflow (two core reviews, inline comments, revisions etc) and this is
a document for developers.

It turns out adding the specs requirement to blueprints adds a
non-trivial amount of overhead, revisit when a blueprint and spec is
needed to minimize overhead in cases where it isn't helpful.

Change-Id: Iaaadc26f3f51ee091419beaa12e0508623eed99e
Fixes network_get_all_by_host to use indexes

The OR in the subquery was constructed in a way that didn't allow
the fixed_ips without instances allocated to be filtered out. It
also had an unnecessary join through virtual interfaces. We have a
denormalized field for instance_uuid with an index in the fixed_ips
table that we can use. This patch removes the unnecessary join and
ads an extra filter inside the OR to reject fixed_ips without
instances. In my test with 300000 fixed ips, this reduced the query
time from about 2.5 seconds to less than 0.01 seconds.

Change-Id: I26fb0dfaf5ff64bd70c947982de4ec1728c84856
Resolves-bug: #1369009
Merge "Fix config generator to use keystonemiddleware"
Merge "Adds LOG statements in multiprocess API test"
Merge "Handle exception better in v2 attach_interface"
Merge "Add InstancePCIRequests object"
Merge "Replace pci_request flavor storage with proper object usage"
Merge "Add request_id in PciDevice"
Merge "Support SR-IOV networking in the PCI modules"
Merge "Support SR-IOV networking in nova compute api and nova neutronv2"
Merge "Support SR-IOV networking in libvirt"
Merge "Clarify logging in lockutils"
Stash numa-related flavor extra_spec items in system_metadata

This is a temporary measure to avoid having to add additional persistence
of extra_specs data simply to facilitate merging the NUMA patches. Only
the numa_cpus.* and numa_mem.* items are saved from extra_specs at
this time. The expectation is that this will quickly come out in kilo
when the rest of the flavor information stashed in system_metadata gets
removed and stored elsewhere.

Blueprint: virt-driver-numa-placement
Change-Id: I83ca863d521c579d8f1fe6168d7b1096af9f14f1
Hook NUMA topology checking into claims

This patch adds the ability to test claims against NUMA topology exposed
by the host and instance. This is currently a no-op as no code in the
scheduler will actually attempt to set the limits against numa_topology.
Since getting the NUMA topology of the instance may require an additional
db roundtrip, we try to avoid getting it unless absolutely necessary.

This patch also changes the resource_tracker resize_claim method to
accept the image_metada of the instance as it is used (alongside flavor)
to calculate the new topology that will be exposed after the resize is
done.

The resource tracker will not do any actual usage tracking yet - this
will be done in a follow-up patch.

Blueprint: virt-driver-numa-placement
Change-Id: I07b690243d481c855c5e706f968cf224b336a1a0
Make resource tracker track NUMA usage

This patch makes the resource tracker actually track and update NUMA
usage from instance andd resize claims, and periodic resource updates.

Change-Id: Iea9d076cb37e530f442faefacd2a59b6b6225e92
Blueprint: virt-driver-numa-placement
API boot process sets NUMA topology for instances

This patch makes the API instance boot path look at the bits that can
contain requested topology, generate the desired topology and make sure
it is saved on the instance itself.

We also need to account for the flavor passed to
VirtNUMAInstanceTopology.get_constraints not being an actual Flavor
object, and work with db layer objects.

Change-Id: Ia3f3446da99a04e2d8152acb7d0bda7a1d502c83
Blueprint: virt-driver-numa-placement
Make HostManager track NUMA usage

This patch adds tracking of NUMA usage in scheduler's HostManager class,
by adding it to the HostState, and making sure it gets updated along
with all the other usage data.

Change-Id: Id31c9a7fd54b918b3d38abb61b305cf7048a6a7d
Blueprint: virt-driver-numa-placement
Add the NUMATopologyFilter

This patch adds the basic NUMA filter that will take the proposed
instance topology and try to match it against that of a host.

The matching will be done in the following manner:

 * Filter will try to match the exact NUMA cells of the instance to those
   of the host. It *will not* attempt to pack the instance onto the host.
 * It will consider the standard over-subscription limits for each cell,
   and provide limits to the compute host accordingly.
 * If instance has no topology defined, it will be considered for any host.
 * If instance has a topology defined, it will be considered only for NUMA
   capable hosts.

DocImpact

Change-Id: I8788dde69524c8a32a41ce31a96c89f9b09e91ce
Blueprint: virt-driver-numa-placement
libvirt: Consider numa_topology when booting

This patch hooks all the already introduced pieces together to enable
NUMA aware placement of instances.

The libvirt driver boot process now looks into the NUMA topology field
of both instance and the host it is being booted on, and based on that
adds some more configuration to the generated guest XML.

In case the host is NUMA capable, but the instance has not requested
a NUMA topology, we will try to pack the instance into a single cell.
If that fails, currently we will not make further attempts (although
an improvement could be to expose the underlying topology to the
instance).

In case the host is NUMA capable, and instance has requested a specific
NUMA topology, we will try to pin the vCPUs of different instance NUMA
cells to the corresponding host NUMA cells, and also expose the instance
NUMA topology to the guest OS.

We will still try to respect CONF.vcpu_pin_set option and avoid pinning
to CPUs that are not in this set.

DocImpact

Change-Id: I611d44ab75475028f63c2b7f8a0d2ef37d85be6e
Blueprint: virt-driver-numa-placement
Updated from global requirements

Change-Id: Ia631e585f9133f4d2f606394a6ffe47c323a9242