preserve order of pre-existing iptables chains

Adds new configuration options:

    iptables_top_regex='' (Default)
        When set treated as a regular expression to match
        iptables rules that should always be placed at the
        top of the table before the nova chains.

    iptables_bottom_regex='' (Default)
        When set treated as a regular expression to match
        iptables rules that should always be placed at the
        bottom of the table right before the COMMIT

Additionally the existing iptables tests were no longer
actually testing the chains we use. In some cases we
were looking for chains that haven't existed in the
code base for 2 years. I took the oppurtunity to update
all tests to actually test for the chains we care about.

Flags: DocImpact

Change-Id: I335ca3712d6dd37051cc8e46e1237aaf66a4a94e
Fixes: bug #1116562
Make scheduler remove dead nodes from its cache.

Scheduler HostManager maintains and updates a cache of all compute
nodes in its host_state_map. However, entries were never deleted from
this mapping, even when the compute nodes were marked as deleted.

This patch removes a state_key from host_state_map when it is not
returned from db.compute_node_get_all, and logs a notice about it.
It also adds unit tests which check for this behaviour.

Change-Id: Ibe6c98dd6c2eb02983db47f263d86cb1b76e1c98
Update tools/regression_tester

* Clarify wording (don't use 'success' and 'failure' instead use 'found'
  and 'no regression test')
* If don't pass gerrit review number run on current branch.  In
  preparation for running via jenkins.
* If no tests were changed, don't run any tests

Change-Id: I5e876e4e5071fc27461d2411fc33db1bf1724405
Add migration with data test for migration 151

Before the 151 migration test is run, _prerun_151 populates the task_log
table with generated date and time in string format for period_beginning and
period_ending. The _check_151 function checks that the migration from string
to DateTime worked as expected.

Partially implements: blueprint migration-testing-with-data

Change-Id: I6d354741a75804bc490ffefde04c5925dd1a2a90
Use joinedload for system_metadata in db

When moving over to nova.db.api not returning sqlalchemy
objects, objects won't be lazy loaded throughout the code, instead the
entire object will be fetched from the DB in nova.db.api.  This change
will trigger any lazyload bugs due to missing joinedload statements
that weren't previously triggered because the lazyloaded data wasn't
being used (and thus fetched from the DB).

Partially implements bp db-api-cleanup

Change-Id: I0dabf3588be0076a6e651c018a1c83ed23e84872
Fix add-fixed-ip and remove-fixed-ip.

There were various places that were passing an integer id from
network api to network manager. On the manager side we need the
uuid so this patch updates everything to send the uuid.

Secondarily, the network info cache wasn't being properly updated
upon return from add-fixed-ip and remove-fixed-ip, so this
changes the code to update the info cache as well.

Fixes bug 1124674

Change-Id: I632b39eb3963998cf42c02584dc9275a8aae243d
Use connection_info on resize

During finish_migration the manager calls initialize_connection but doesn't
update the block_device_mapping with the (potentially new) connection_info
returned.

This can cause the bdm to get out of sync, the volume to fail to attach, and
the resize to fail.

Fixes bug #1076801

Change-Id: Ie49ccd2138905e178843b375a9b16c3fe572d1db
Only dhcp the first ip for each mac address.

When using add-fixed-ip there can be more than one ip for each
mac so make dnsmasq just serve the first one. Also, make sure to
disassociate the fixed ips when they are removed since dnsmasq
will never know about them.

Fixes bug 1124692

Change-Id: If41c03e1adb408ce28d8e8a25706ebc76fb3c4e1
Synchronize code from oslo

Use commit eaab5fae2502198e9fa57d0d90a7204a2bd83b16:
Merge "sort options to make --help output prettier"
(Wed Feb 13 12:52:14 2013 +0000).

Changes:
  9669767 Fix PEP8 error in oslo-rootwrap
  e3e5e0e Fixes "is not", "not in" syntax usage
  d156150 Implements import_group
  0ce65aa sort options to make --help output prettier
  580c259 Make tox run doctests
  d8c4e0c Change Exception MissingArgs's string
  6d102bc Provide i18n to those messages without _()
  cf705c5 Make project pyflakes clean
  9e5912f Fix pep8 E125 errors
  4a1ec21 Support testing args for LocalhostMatchMaker
  9fd6437 Exchanges should return directed topics
  a4b6c31 Allow running test in uninstalled source tree
  1461135 timeutils: considers that now is soon
  a956f7a Import timeutils.is_soon from keystoneclient
  a4b6c31 Allow running test in uninstalled source tree
  076e9e5 Add support for directly stringifying VersionInfo

Change-Id: I427508f0882a528d040c89290ff9ca68a1e91bcd
Fixes: bug #1124213
Don't modify injected_files inside PXE driver

Make a local copy of injected_files so that we don't modify the
original.

Change-Id: I508cfc0baec15a091a1717eb2aebb9dd39278e5b
add config drive to api_samples

Add config drive extension show and details to api_samples

blueprint nova-api-samples
bug 1091737

Change-Id: I6ae1f01a417f153723b85d08c2f181909c154946
Remove a log message in test code.

This log message indicates that this code should only run during
testing.  The downside is that it does run during testing and can cause
a bunch of annoying noise in test logs.  Just remove it.

Change-Id: I2cc18f3b7871dbfa8859e33225fb7c2bb1ccc53b
relocatable roots doesn't handle testr args/opts

Previous fix (4c891b9243985bb95d8beff8affa9db470a1c94a)
changed how the args are parsed but didn't update the logic
for handling testr args/opts. As a result args/opts for testr
where not be properly passed.

This fixes corrects that oversite and now properly parses those
args as well.

Change-Id: I418b0478b2a180e497d041fcab25158c52f44bac
Fixes: #1116942
Add option to allow cross AZ attach configurable

Make check_attach() optionally check if the volume and instance
are in the same availability zone and if cross AZ attach
is configured as not allowed report error.

This does not change the current default behaviour.

DocImpact: Adds a new Nova Config option

Change-Id: Ib0e085888b1c6620869261d87cd964de302accb3
use nova-conductor for live-migration

The function check_can_live_migrate_source in compute/manager.py
attempts to make a direct DB query. This small change makes the
query through the conductor service

Change-Id: I9edc0aec89b4ce030f20020204e7c872f2f4fbd8
fixes: bug 1124594
filter_scheduler: Select from a subset of hosts.

Adding a subset of best hosts to choose a host from, instead
of choosing the host with the highest weight. The subset size is
configurable. Setting the subset size to 1 will mimic the old
behavior.

DocImpact

Change-Id: Icc6f8a7ca7cf992d99b872a45e241f13fa6e1536
Fixes: bug 1098635
Add an update option to run_tests.sh

This commit adds an update option to run_tests.sh. This option
is used to rerun install_venv.py on an already installed venv.
This will then just update out of date packages with pip.

Change-Id: Iac065f71a64a6a7f7f2ee000ee01be9f78af6754
Adds API Sample tests for FlavorAccess extension.

Partially implements blueprint nova-api-samples
Fixes bug 1091741

Change-Id: I65bb9e508996417bdcdc9ec87997df5373c06bad
Allow archiving deleted rows to shadow tables, for performance.

Implements bp db-archiving

Add versions of all tables with "shadow_" prepended to their names.
These shadow tables do not have indexes.

Adds API function archive_deleted_rows, which moves some deleted
rows from production tables to the corresponding shadow tables.
(It only does a few rows, to avoid blocking the process or hitting
the database too hard.  So it needs to be called regularly.)

Also adds a db archive_deleted_rows command to nova-manage.

Uses Boris Pavlovic's code for copying columns from existing tables
to new tables.

Change-Id: I7c8f2fb8794b6f9cd2531e54a1a9635010f8c97e
[xenapi] Cooperatively yield during sparse copy

On a resize down the sparse copy blocks other operations from running on
the compute node, and can take a long time to finish.  This patch adds a
greenthread yield to not block other things from happening on the
compute node.

Change-Id: I1287a6a0bb3157ad0e5eafe53b02db21849650fa