Fixes a race condition on updating security group rules

During creating the instance, compute manager makes an rpc call
to "allocate_for_instance" from the network manager, which will
in turn end up making a rpc cast to each compute_manager with
an instance in the group(s) than need to be refreshed calling
refresh_instance_security_rules.

When an instance is being created is in a group that has a
rule referring to itself, that chain will end up trying to
refresh the rules for the instance which is still being created.

That call will find its way down to do_refresh_instance_rules()
in libvirt/firewall - which will in turn try to get network_info
from the global "network_infos" in firewall.

But the entry in network_infos for this instance might not of
been added yet, as that only happens when the create of the
instance calls "prepare_instance_filter which comes later in
the sequence as part of driver.spawn

The fix here adds a synchronisation lock to the incomming
refresh call so that it is blocked until any in-progress
thread on the instance is finished.

Change-Id: I550e286aa4777b011633550109e18e03d63a4bbb
preserve order of pre-existing iptables chains

Adds new configuration options:

    iptables_top_regex='' (Default)
        When set treated as a regular expression to match
        iptables rules that should always be placed at the
        top of the table before the nova chains.

    iptables_bottom_regex='' (Default)
        When set treated as a regular expression to match
        iptables rules that should always be placed at the
        bottom of the table right before the COMMIT

Additionally the existing iptables tests were no longer
actually testing the chains we use. In some cases we
were looking for chains that haven't existed in the
code base for 2 years. I took the oppurtunity to update
all tests to actually test for the chains we care about.

Flags: DocImpact

Change-Id: I335ca3712d6dd37051cc8e46e1237aaf66a4a94e
Fixes: bug #1116562
Update tools/regression_tester

* Clarify wording (don't use 'success' and 'failure' instead use 'found'
  and 'no regression test')
* If don't pass gerrit review number run on current branch.  In
  preparation for running via jenkins.
* If no tests were changed, don't run any tests

Change-Id: I5e876e4e5071fc27461d2411fc33db1bf1724405
Add migration with data test for migration 151

Before the 151 migration test is run, _prerun_151 populates the task_log
table with generated date and time in string format for period_beginning and
period_ending. The _check_151 function checks that the migration from string
to DateTime worked as expected.

Partially implements: blueprint migration-testing-with-data

Change-Id: I6d354741a75804bc490ffefde04c5925dd1a2a90
Fix add-fixed-ip and remove-fixed-ip.

There were various places that were passing an integer id from
network api to network manager. On the manager side we need the
uuid so this patch updates everything to send the uuid.

Secondarily, the network info cache wasn't being properly updated
upon return from add-fixed-ip and remove-fixed-ip, so this
changes the code to update the info cache as well.

Fixes bug 1124674

Change-Id: I632b39eb3963998cf42c02584dc9275a8aae243d
Use connection_info on resize

During finish_migration the manager calls initialize_connection but doesn't
update the block_device_mapping with the (potentially new) connection_info
returned.

This can cause the bdm to get out of sync, the volume to fail to attach, and
the resize to fail.

Fixes bug #1076801

Change-Id: Ie49ccd2138905e178843b375a9b16c3fe572d1db
Only dhcp the first ip for each mac address.

When using add-fixed-ip there can be more than one ip for each
mac so make dnsmasq just serve the first one. Also, make sure to
disassociate the fixed ips when they are removed since dnsmasq
will never know about them.

Fixes bug 1124692

Change-Id: If41c03e1adb408ce28d8e8a25706ebc76fb3c4e1
Synchronize code from oslo

Use commit eaab5fae2502198e9fa57d0d90a7204a2bd83b16:
Merge "sort options to make --help output prettier"
(Wed Feb 13 12:52:14 2013 +0000).

Changes:
  9669767 Fix PEP8 error in oslo-rootwrap
  e3e5e0e Fixes "is not", "not in" syntax usage
  d156150 Implements import_group
  0ce65aa sort options to make --help output prettier
  580c259 Make tox run doctests
  d8c4e0c Change Exception MissingArgs's string
  6d102bc Provide i18n to those messages without _()
  cf705c5 Make project pyflakes clean
  9e5912f Fix pep8 E125 errors
  4a1ec21 Support testing args for LocalhostMatchMaker
  9fd6437 Exchanges should return directed topics
  a4b6c31 Allow running test in uninstalled source tree
  1461135 timeutils: considers that now is soon
  a956f7a Import timeutils.is_soon from keystoneclient
  a4b6c31 Allow running test in uninstalled source tree
  076e9e5 Add support for directly stringifying VersionInfo

Change-Id: I427508f0882a528d040c89290ff9ca68a1e91bcd
Fixes: bug #1124213
relocatable roots doesn't handle testr args/opts

Previous fix (4c891b9243985bb95d8beff8affa9db470a1c94a)
changed how the args are parsed but didn't update the logic
for handling testr args/opts. As a result args/opts for testr
where not be properly passed.

This fixes corrects that oversite and now properly parses those
args as well.

Change-Id: I418b0478b2a180e497d041fcab25158c52f44bac
Fixes: #1116942
Add option to allow cross AZ attach configurable

Make check_attach() optionally check if the volume and instance
are in the same availability zone and if cross AZ attach
is configured as not allowed report error.

This does not change the current default behaviour.

DocImpact: Adds a new Nova Config option

Change-Id: Ib0e085888b1c6620869261d87cd964de302accb3
Fix script argument parsing.

The vm_vdi_cleaner.py script's argument parsing was broken and unloved.

Change-Id: I1662534d65fd0e8472e092ed9f10d6cc059f1cab
Adds API Sample tests for FlavorAccess extension.

Partially implements blueprint nova-api-samples
Fixes bug 1091741

Change-Id: I65bb9e508996417bdcdc9ec87997df5373c06bad
Harmonize PEP8 checking between tox and run_tests.sh

Tox and run_tests.sh were running PEP8 checks against different
file-sets. This patch refactors the logic to determine which files to
run PEP8 checks on into `tools/run_pep8.sh` where it can be called by
both tox and `run_tests.sh`.

Additional fixes:

Some of our Python XenAPI Dom0 plugins don't end in *.py but should
still be checked by PEP8. This patches fixes the hacking.py violations
in the files and adds them back to the srcfiles list.

Merged tools/unused_imports.sh into tools/run_pep8.sh

Change-Id: Id5edd1acb644ab938beffc3473494a179d9d8cda
Delete baremetal interfaces when their parent node is deleted

Change-Id: I504cbed0adea86cab840c40ea9b9f96faf299f73
Merge "Harmonize PEP8 checking between tox and run_tests.sh"
Remove dead variable assignment in compute manager

The assigement of 'db_power_state' in '_sync_power_state' is
never used before it is overwritten a few lines later

Blueprint: compute-driver-events
Change-Id: I0a7e88fc7c3067a09b0a05a643ae4e6d40a11488
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Split out body of loop in _sync_power_states in compute manager

The _sync_power_states method in the compute manager iterates
over all instances in the DB, synchronizing their power
states. The introduction of async events to the compute
drivers requires a method for synchronizing the power state
of a single instance at a time. Split out the body of the
loop in _sync_power_states to a method _sync_instance_power_state

Blueprint: compute-driver-events
Change-Id: I5eee72b39f765a2444388005fabba2cff7ff4b29
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Ensure start time is earlier than end time in simple_tenant_usage

Simple check and respond with a http 400 if the start time is later

Fixes LP# 1118910

Change-Id: I95f12abca725f347488e620d8c699f424e53ed23
Add image to request_spec during resize

Image is needed by ImagePropertiesFilter during scheduling.

Resolves bug 1117882.

Change-Id: Icfbba85e2fdeb609d5fdabb6a7c3167505fe23b0
Speedup the revert_state test

test_revert_state can take over 10 seconds, but more than half of the
time is spent in run_instance() which doesn't change the result.
Remove the call to speed up the tests.

Change-Id: I0557d6822576b99376da8691ee6e76b759d3bc9c