Make scheduler remove dead nodes from its cache.

Scheduler HostManager maintains and updates a cache of all compute
nodes in its host_state_map. However, entries were never deleted from
this mapping, even when the compute nodes were marked as deleted.

This patch removes a state_key from host_state_map when it is not
returned from db.compute_node_get_all, and logs a notice about it.
It also adds unit tests which check for this behaviour.

Change-Id: Ibe6c98dd6c2eb02983db47f263d86cb1b76e1c98
Use joinedload for system_metadata in db

When moving over to nova.db.api not returning sqlalchemy
objects, objects won't be lazy loaded throughout the code, instead the
entire object will be fetched from the DB in nova.db.api.  This change
will trigger any lazyload bugs due to missing joinedload statements
that weren't previously triggered because the lazyloaded data wasn't
being used (and thus fetched from the DB).

Partially implements bp db-api-cleanup

Change-Id: I0dabf3588be0076a6e651c018a1c83ed23e84872
Don't modify injected_files inside PXE driver

Make a local copy of injected_files so that we don't modify the
original.

Change-Id: I508cfc0baec15a091a1717eb2aebb9dd39278e5b
add config drive to api_samples

Add config drive extension show and details to api_samples

blueprint nova-api-samples
bug 1091737

Change-Id: I6ae1f01a417f153723b85d08c2f181909c154946
Remove a log message in test code.

This log message indicates that this code should only run during
testing.  The downside is that it does run during testing and can cause
a bunch of annoying noise in test logs.  Just remove it.

Change-Id: I2cc18f3b7871dbfa8859e33225fb7c2bb1ccc53b
use nova-conductor for live-migration

The function check_can_live_migrate_source in compute/manager.py
attempts to make a direct DB query. This small change makes the
query through the conductor service

Change-Id: I9edc0aec89b4ce030f20020204e7c872f2f4fbd8
fixes: bug 1124594
filter_scheduler: Select from a subset of hosts.

Adding a subset of best hosts to choose a host from, instead
of choosing the host with the highest weight. The subset size is
configurable. Setting the subset size to 1 will mimic the old
behavior.

DocImpact

Change-Id: Icc6f8a7ca7cf992d99b872a45e241f13fa6e1536
Fixes: bug 1098635
Add an update option to run_tests.sh

This commit adds an update option to run_tests.sh. This option
is used to rerun install_venv.py on an already installed venv.
This will then just update out of date packages with pip.

Change-Id: Iac065f71a64a6a7f7f2ee000ee01be9f78af6754
Allow archiving deleted rows to shadow tables, for performance.

Implements bp db-archiving

Add versions of all tables with "shadow_" prepended to their names.
These shadow tables do not have indexes.

Adds API function archive_deleted_rows, which moves some deleted
rows from production tables to the corresponding shadow tables.
(It only does a few rows, to avoid blocking the process or hitting
the database too hard.  So it needs to be called regularly.)

Also adds a db archive_deleted_rows command to nova-manage.

Uses Boris Pavlovic's code for copying columns from existing tables
to new tables.

Change-Id: I7c8f2fb8794b6f9cd2531e54a1a9635010f8c97e
[xenapi] Cooperatively yield during sparse copy

On a resize down the sparse copy blocks other operations from running on
the compute node, and can take a long time to finish.  This patch adds a
greenthread yield to not block other things from happening on the
compute node.

Change-Id: I1287a6a0bb3157ad0e5eafe53b02db21849650fa
VirtualPowerDriver catches ProcessExecutionError.

Baremetal virtual_power_driver now catches ProcessExecutionError coming
from ssh_execute and logs an exception. Requested actions (eg, power_on)
return an error state instead of raising an exception.

Change-Id: I8f66ba843e12de13f9a14d736f7bef2eda4ef85c
Merge "Remove a log message in test code."
Merge "Use joinedload for system_metadata in db"
Merge "add config drive to api_samples"
Merge "[xenapi] Cooperatively yield during sparse copy"
Merge "Add an update option to run_tests.sh"
Merge "filter_scheduler: Select from a subset of hosts."
Merge "Make scheduler remove dead nodes from its cache."
Merge "VirtualPowerDriver catches ProcessExecutionError."
Merge "Allow archiving deleted rows to shadow tables, for performance."