preserve order of pre-existing iptables chains

Adds new configuration options:

    iptables_top_regex='' (Default)
        When set treated as a regular expression to match
        iptables rules that should always be placed at the
        top of the table before the nova chains.

    iptables_bottom_regex='' (Default)
        When set treated as a regular expression to match
        iptables rules that should always be placed at the
        bottom of the table right before the COMMIT

Additionally the existing iptables tests were no longer
actually testing the chains we use. In some cases we
were looking for chains that haven't existed in the
code base for 2 years. I took the oppurtunity to update
all tests to actually test for the chains we care about.

Flags: DocImpact

Change-Id: I335ca3712d6dd37051cc8e46e1237aaf66a4a94e
Fixes: bug #1116562
Update tools/regression_tester

* Clarify wording (don't use 'success' and 'failure' instead use 'found'
  and 'no regression test')
* If don't pass gerrit review number run on current branch.  In
  preparation for running via jenkins.
* If no tests were changed, don't run any tests

Change-Id: I5e876e4e5071fc27461d2411fc33db1bf1724405
Add migration with data test for migration 151

Before the 151 migration test is run, _prerun_151 populates the task_log
table with generated date and time in string format for period_beginning and
period_ending. The _check_151 function checks that the migration from string
to DateTime worked as expected.

Partially implements: blueprint migration-testing-with-data

Change-Id: I6d354741a75804bc490ffefde04c5925dd1a2a90
Fix add-fixed-ip and remove-fixed-ip.

There were various places that were passing an integer id from
network api to network manager. On the manager side we need the
uuid so this patch updates everything to send the uuid.

Secondarily, the network info cache wasn't being properly updated
upon return from add-fixed-ip and remove-fixed-ip, so this
changes the code to update the info cache as well.

Fixes bug 1124674

Change-Id: I632b39eb3963998cf42c02584dc9275a8aae243d
Use connection_info on resize

During finish_migration the manager calls initialize_connection but doesn't
update the block_device_mapping with the (potentially new) connection_info
returned.

This can cause the bdm to get out of sync, the volume to fail to attach, and
the resize to fail.

Fixes bug #1076801

Change-Id: Ie49ccd2138905e178843b375a9b16c3fe572d1db
Only dhcp the first ip for each mac address.

When using add-fixed-ip there can be more than one ip for each
mac so make dnsmasq just serve the first one. Also, make sure to
disassociate the fixed ips when they are removed since dnsmasq
will never know about them.

Fixes bug 1124692

Change-Id: If41c03e1adb408ce28d8e8a25706ebc76fb3c4e1
Synchronize code from oslo

Use commit eaab5fae2502198e9fa57d0d90a7204a2bd83b16:
Merge "sort options to make --help output prettier"
(Wed Feb 13 12:52:14 2013 +0000).

Changes:
  9669767 Fix PEP8 error in oslo-rootwrap
  e3e5e0e Fixes "is not", "not in" syntax usage
  d156150 Implements import_group
  0ce65aa sort options to make --help output prettier
  580c259 Make tox run doctests
  d8c4e0c Change Exception MissingArgs's string
  6d102bc Provide i18n to those messages without _()
  cf705c5 Make project pyflakes clean
  9e5912f Fix pep8 E125 errors
  4a1ec21 Support testing args for LocalhostMatchMaker
  9fd6437 Exchanges should return directed topics
  a4b6c31 Allow running test in uninstalled source tree
  1461135 timeutils: considers that now is soon
  a956f7a Import timeutils.is_soon from keystoneclient
  a4b6c31 Allow running test in uninstalled source tree
  076e9e5 Add support for directly stringifying VersionInfo

Change-Id: I427508f0882a528d040c89290ff9ca68a1e91bcd
Fixes: bug #1124213
Don't modify injected_files inside PXE driver

Make a local copy of injected_files so that we don't modify the
original.

Change-Id: I508cfc0baec15a091a1717eb2aebb9dd39278e5b
relocatable roots doesn't handle testr args/opts

Previous fix (4c891b9243985bb95d8beff8affa9db470a1c94a)
changed how the args are parsed but didn't update the logic
for handling testr args/opts. As a result args/opts for testr
where not be properly passed.

This fixes corrects that oversite and now properly parses those
args as well.

Change-Id: I418b0478b2a180e497d041fcab25158c52f44bac
Fixes: #1116942
Add option to allow cross AZ attach configurable

Make check_attach() optionally check if the volume and instance
are in the same availability zone and if cross AZ attach
is configured as not allowed report error.

This does not change the current default behaviour.

DocImpact: Adds a new Nova Config option

Change-Id: Ib0e085888b1c6620869261d87cd964de302accb3
use nova-conductor for live-migration

The function check_can_live_migrate_source in compute/manager.py
attempts to make a direct DB query. This small change makes the
query through the conductor service

Change-Id: I9edc0aec89b4ce030f20020204e7c872f2f4fbd8
fixes: bug 1124594
Adds API Sample tests for FlavorAccess extension.

Partially implements blueprint nova-api-samples
Fixes bug 1091741

Change-Id: I65bb9e508996417bdcdc9ec87997df5373c06bad
Allow archiving deleted rows to shadow tables, for performance.

Implements bp db-archiving

Add versions of all tables with "shadow_" prepended to their names.
These shadow tables do not have indexes.

Adds API function archive_deleted_rows, which moves some deleted
rows from production tables to the corresponding shadow tables.
(It only does a few rows, to avoid blocking the process or hitting
the database too hard.  So it needs to be called regularly.)

Also adds a db archive_deleted_rows command to nova-manage.

Uses Boris Pavlovic's code for copying columns from existing tables
to new tables.

Change-Id: I7c8f2fb8794b6f9cd2531e54a1a9635010f8c97e
Harmonize PEP8 checking between tox and run_tests.sh

Tox and run_tests.sh were running PEP8 checks against different
file-sets. This patch refactors the logic to determine which files to
run PEP8 checks on into `tools/run_pep8.sh` where it can be called by
both tox and `run_tests.sh`.

Additional fixes:

Some of our Python XenAPI Dom0 plugins don't end in *.py but should
still be checked by PEP8. This patches fixes the hacking.py violations
in the files and adds them back to the srcfiles list.

Merged tools/unused_imports.sh into tools/run_pep8.sh

Change-Id: Id5edd1acb644ab938beffc3473494a179d9d8cda
Merge "Allow archiving deleted rows to shadow tables, for performance."
Merge "use nova-conductor for live-migration"
Merge "Don't modify injected_files inside PXE driver"
Merge "Harmonize PEP8 checking between tox and run_tests.sh"
Remove dead variable assignment in compute manager

The assigement of 'db_power_state' in '_sync_power_state' is
never used before it is overwritten a few lines later

Blueprint: compute-driver-events
Change-Id: I0a7e88fc7c3067a09b0a05a643ae4e6d40a11488
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Split out body of loop in _sync_power_states in compute manager

The _sync_power_states method in the compute manager iterates
over all instances in the DB, synchronizing their power
states. The introduction of async events to the compute
drivers requires a method for synchronizing the power state
of a single instance at a time. Split out the body of the
loop in _sync_power_states to a method _sync_instance_power_state

Blueprint: compute-driver-events
Change-Id: I5eee72b39f765a2444388005fabba2cff7ff4b29
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>