preserve order of pre-existing iptables chains

Adds new configuration options:

    iptables_top_regex='' (Default)
        When set treated as a regular expression to match
        iptables rules that should always be placed at the
        top of the table before the nova chains.

    iptables_bottom_regex='' (Default)
        When set treated as a regular expression to match
        iptables rules that should always be placed at the
        bottom of the table right before the COMMIT

Additionally the existing iptables tests were no longer
actually testing the chains we use. In some cases we
were looking for chains that haven't existed in the
code base for 2 years. I took the oppurtunity to update
all tests to actually test for the chains we care about.

Flags: DocImpact

Change-Id: I335ca3712d6dd37051cc8e46e1237aaf66a4a94e
Fixes: bug #1116562
Fix inaccuracies in the development environment doc.

Change-Id: If2b0884de54b75e40896cee9b4fb5eb805de2624
Ensure 'subunit2pyunit' is run in venv from run_tests.sh

The run_tests.sh script invokes

  bash -c "${wrapper} $TESTRTESTS | subunit2pyunit"

where '${wrapper}' expands to 'tools/with_venv.sh'. Due to shell
precedence rules, this only applies to '$TESTRTESTS' not the
'subunit2pyunit' part of the command. The 'subunit2pyunit' binary
though may only be available from the .venv

Change-Id: I5972c8f77b3f50a3713881e6ca1acbaeabbd47d4
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Fix regression in non-admin simple_usage:show.

Commit 6ff32210772a67a1b526d9d784030afc90f3ce99 optimized the db query
for usage but replaced a call that had require_context with one that
had require_admin_context. This causes usage show to throw a 403. We
check for admin_or_owner in policy, so there is no need to check for
admin context at the db layer.

Fixes bug 1122267

Change-Id: Iff362f46d83a710f3883538bcb646e3457e0ba64
Merge "support reloctable venv roots in testing framework"
Fix unconsumed column name warning in test_migrations.

The warning is "SAWarning: Unconsumed column names: aggregate_name"

There is no column aggregate_name in the aggregates table.

Change-Id: I3b92b3429ac3e020b9ab3f7ce10c73a4e4b93187
Fixes a bug in attaching volumes on Hyper-V

Fixes bug: #1122264

The CIM v2 WMI context needs to be instantiated by the BaseVolumeUtils
class and not passed by the caller of get_iscsi_initiator().

Change-Id: I811b5bd0117b0035c18e5c90169b25a095bc54e9
Refine PowerVM MAC address generation algorithm

Change MAC address generation algorithm on PowerVM to generate
an additional 4 bits of data and improve comments around it.

Change-Id: Icdfba54161bcb043c17127c26c57503af497d001
Use ServiceGroup API to show node liveness

Previously it's direct db access. Now we use the servicegroup
API to show node liveness. Note that db access is still needed
to get other service attributes such as `disabled'.

Change-Id: I65ef3a136fd90036a8d009ed97a5f48ae461d8d9
Remove locking declarator in ServiceGroup __new__()

The lock-based synchronization is not re-entrant, which
would introduce a deadlock with the new conductor patch.
Because race condition is not be possible in the
eventlet threading model with GIL, we simply remove the
synchronized declarator.

Change-Id: I03e52227385dafd2b2b66bca18cab8445c91f3be
Add generic UC dropper

There is a lot of tables where we should add unique constraints
on upgrade and drop on downgrade, to avoid race conditions.
But there is a problem with SQLite. SQLite does not allow to drop unique constraints.
There is only one way to drop UC in SQLite:
1) create new table, with same columns and UC (but without UC that we want to drop)
2) copy data from old table
3) drop old table
4) create indexes in new table from old
5) rename new table to old table name

For other DB backends is used 'alter table'.

This patch add generic method for dropping UC.

blueprint db-unique-keys

Change-Id: I3e60578d3902559996943868abeb17b41153769b
Allow fixed to float ping with external gateway.

If you ping an a floating ip from an instance with only a fixed ip, the
traffic will be sent to the default gateway. If the default gateway has
a route to the floating ip, it will then be sent to the host of the
instance with the floating ip. The source address will be on the fixed
network, so if the floating addresses are on a different network, the
receiving host to drop the packet due to the default rp_filter.
Essentially, the route for the fixed range is on a different interface
so it the kernel assumes the packet is spoofed and drops it.

In order to fix this issue this patch adds a new config option called
force_snat_range. Traffic that originates in fixed_range and has a
destination in force_snat_range will always be routed. This forces
the packet to hit the fallback floating roule and be snatted to
routing_source_ip. This means the traffic will originate from the
proper network and the packet will make it through.

DocImpact

Fixes bug 1122335

Change-Id: I7e31c25d37a5c4fd02c8238aec58a28af3fd7f6a
More conductor support for resizes

This patch adds more support for resizes to conductor, which apparently
are not getting tested in the "full" tempest gate.

This adds network_migrate_instance_{start,finish}() calls to conductor,
which offloads the network_api calls of similar name over RPC for
components that can't take the DB hit of calling network_api directly.
Also, in quota, the commit and rollback calls hit the database, so
export those from conductor as well.

This is likely the source of the failures in bug 1119817

Change-Id: I56bb65b0e168e36e138cf6bf125d6e83bf71f50b
Make scheduler remove dead nodes from its cache.

Scheduler HostManager maintains and updates a cache of all compute
nodes in its host_state_map. However, entries were never deleted from
this mapping, even when the compute nodes were marked as deleted.

This patch removes a state_key from host_state_map when it is not
returned from db.compute_node_get_all, and logs a notice about it.
It also adds unit tests which check for this behaviour.

Change-Id: Ibe6c98dd6c2eb02983db47f263d86cb1b76e1c98
test_(dis)associate_by_non_existing_security_group_name missing stub

test_(dis)associate_by_non_existing_security_group_name() is passing because
it is not finding the instance which also returns HTTPNotFound not because
it cannot find the security group. This patch adds the missing stub so
that it it checks for the non existing security group.

Fixes bug 1120832

Change-Id: I67a2b82ac329193c2438038719f8c052de8081a3
Implements resize / cold migration on Hyper-V

Blueprint: hyper-v-compute-resize

Resize / cold migration is implemented by copying the local disks to a remote
SMB share, identified by the configuration option HYPERV.instances_path_share
or, if empty, by an administrative share with a remote path corresponding to
the configuration option instances_path.

The source instance directory is renamed by adding a suffix "_revert" and
preserved until the migration is confirmed or reverted. In the former case
the directory will be deleted and in the latter renamed to the original name.

The VM corresponding to the instance is deleted on the source host and
recreated on the target. Any mapped volume is disconnected on the source
and reattached to the new VM on the target host.

In case of resize operations, the local VHD file is resized according to
the new flavor limits. Due to VHD limitations, an attempt to resize a disk
to a smaller size will result in an exception.

In case of differencing disks (CoW), should the base disk be missing
in the target host's cache, it will be downloaded and reconnected to the
copied differencing disk.

Same host migrations are supported by using a temporary directory
with suffix "_tmp" during disk file copy.

Unit tests have been added for the new features accordingly.

Change-Id: Ieee2afff8061d2ab73a2252b7d2499178d0515fd
Allow generic rules in context_is_admin rule in policy.

context_is_admin role is used by nova to check if
the current user is the admin. But it can only check
role rules. The fix allow generic rules in context_is_admin.

DocImpact

Fixes bug 1118142

Change-Id: Ib4823a67fe63d5356fc8c9280a2013b8855f5217
Update docs about testing.

Add an entry to the HACKING file about testr. While in there, noticed a
reference to the now-defunct nova/testing dir. Fixed that, moved the testing
README into nova/tests and remove the nova/testing dir.

Change-Id: Ibf6fb82658ba73eee9123fa53b340d0b72afb292
Merge "Implement ZooKeeper driver for ServiceGroup API."
Merge "Fix boto capabilities check"