sync oslo log updates

this helps address the issue where syslog gets overloaded and we
loose the ability to figure out what's going on in clients like
nova-manage

Change-Id: Iedde5ed4e8e23e204be4fa7a19e9514acf2ebf86
Run libguestfs API calls in a thread pool

The libguestfs API is a binding to a native C library, and thus
it will block greenthreads. To avoid this all libguestfs API
calls need to be run in a threadpool. This can be done by just
wrapping the GuestFS object in an eventlet.tpool.Proxy object

Fixes bug #1132742
Change-Id: I877ad9e88442e41c9ad234503d9b1ddf1fb27408
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Remove unused nova.db.api:instance_get_all_by_reservation

Change-Id: I55192773eaa18eb81ffb2b5ec3567b09116b0d1e
Unpin routes dependency version

Fixes bug #1133258

It looks like we should be fine with 1.13 and have no reason to expect
future versions will break us.

Cinder and Quantum both require >=1.12.3 too.

Change-Id: Ice0649d98e2a46505503072c07509fe274f195b6
Unpin PasteDeploy dependency version

Fixes bug #1133262

There are no newer releases of PasteDeploy than 1.5.0 and we have no
reason to believe newer versions will break us. Indeed, if an
incompatible version is released, we will want to know about it quickly.

Change-Id: I331b33b16ea50202886b19bf5c4060d9fa46d5ec
Fix crash in quantumapi if no network or port id is specified

The existing code assumes that if no port id or fixed ip was specified,
that the network id passed in is valid. This is not necessarily the case,
and results in adding None to the list of network ids, which later
causes a check to see if the network id list is empty to fail.

This only adds the network id to the list if it was specified, and
adds a more explicit check to the region that previously fell through
without finding a matching network so that a similar case would be
easier to debug in the future.

Addresses bug #1131264

Change-Id: I7a8aaea5772707b8bd6313249e74b1302eedefe8
Make allocate_for_instance() return only info about ports allocated

Previously, the compute manager was filtering the results based on the
port id that the user requested. However, if they do not pass one (which
is valid) then we can not do the filter and we fail. This makes the
allocation function do the filtering, which should not affect the normal
use case where it is only called once, since all the ports returned were
allocated at that time.

The manager filter behavior is thus removed and a test is added to
verify that the allocate method does the right thing the second time
around.

This fixes the fallout found after fixing the actual cause of bug 1131264

Change-Id: I8f3af96051268503596007491af365e8ce28f5b3
Use min_ram of original image for snapshot, even with VHD

When snapshotting an instance there seems to be no reason that min_ram
can't be set based off of the original image in all cases.  This relaxes
the restriction that VHD backed instances set min_ram for snapshots
based on the current instance.

Bug 1133524

Change-Id: I3779c48d7325f3f0bde496f43a70017fe79d2835
Make sure no duplicate forward rules can exist.

Some failure conditions can cause floating forwards to not be properly
removed. We know that duplicate floating rules will never work, so
even if we have had other failures there is no need to create duplicate
forward rules. This patch fixes the issue by removing all existing
forwards for a floating ip before adding new ones. It adds a method
to IptablesTable to remove rules by regex and includes tests for new
method and the specific case of a duplicate forward.

Fixes bug 1133527

Change-Id: Iaf5df8406fc49e92bac328cd81ffda1969b77d68
Improve hackings docstring detection

Previously: def foo():\n    a = '''This is not a docstring'''
would be counted as a docstring even though it isn't.

Fix this and add docstring tests.

Fix bug 1100912

Change-Id: Id4d4aa382713eea340720e166e418262ba68eaaf
Add a 'hw_' namespace to glance hardware config properties

This updates the libvirt driver to use a 'hw_' prefix on
the names of hardware config properties it looks up for
glance images. Thus the new syntax for setting custom
hardware properties is

      # glance image-update \
               --property hw_disk_bus=ide \
               --property hw_cdrom_bus=ide \
               --property hw_vif_model=e1000 \
               f16-x86_64-openstack-sda

There is no back-compat support for the previous 'disk_bus',
'cdrom_bus' and 'vif_model' names, since they were never
made available in any formal release, only development
snapshots.

DocImpact
Blueprint: libvirt-custom-hardware
Fixes bug 1132739
Change-Id: Ic15ef0eba065113b35bbb2713be2532c858c3534
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Improve output msgs for _compare_result

In case of errors, the messages are now more detailed.
Fixes bug 1133621

Change-Id: I7747d7c4dcaf2199dc55139ee0235f6adca1ad36
Prevent hacking.py from crashing on unexpected import exception

If a module isn't importable due to an unexpected error during
__import__.  Ignore it, hacking.py is only a style checker.

Fix bug 1134622

Change-Id: I704b7cc9a7bd415018ae7634767b7bd9c3b7d9b2
ec2 _format_security_group() accesses db when using quantum_driver

This patch fixes _format_security_group() so that it does not
access the db when using quantum. In addition, this patch implements
the querying params that were missing in get() and list() in the quantum
security group driver.

Fixes bug 1131956

Change-Id: I5ec65829301bca7e5b215d5052a18528dff76517
Fix `with_data` handling in test-migrations.

Two issues with the existing code:

1) migrate-down wasn't respecting the `with_data` flag

2) migrate-up was not actually migrating unless `with_data` was True.
The correct behavior is to always migrate up, but only run checks when
`with_data` is set.

Change-Id: I781fa4796908bea684d4499fa9ff5e4d7a18fbd4
Suppress retries on UnexpectedTaskStateErrors

Don't try to re-schedule if an UnexpectedTaskStateError occurs during
a build.  If the instance was deleted mid-build, just catch the
exception and log a message.

Change-Id: I5e577f252ee71f310a749186060808723b676854
Don't blindly skip first migration.

The existing code would, no matter-what, skip the first migration when
running through the migrations tests. This makes the code
slightly counterintuitive (why skip the first one?), and is not very
flexible.

The reworked logic lets a migration raise a `NotImplementedError` to
signal that it doesn't support a `downgrade`. This lets the
walk-versions code treat the first-migration just like any other.

Change-Id: I19fd8ebc6075392cf9e573b8b4166291584b4a83
Remove old commented out code in sqlalchemy models

Change-Id: I0e49d00ce8790c8498976c2bf7bcfd9b69bfb5a4
Shrink size of aggregate_metadata_get_by_host sql query

* Remove unnecessary secondary joins in Aggregate model.
* Remove unnecessary backrefs (only needed to create bidirectional relationship)

aggregate_metadata_get_by_host goes from 6 JOINS down to two JOINS.

Fix bug 1133495

Change-Id: I72966fa205973ec638816b07bfdcd54f1102feb5
Remove duplicate options(joinedload) from aggregates db code

_aggregate_get_query already does option joinedload.  So when use it don't
need to use additional options(joinedload).

Change-Id: I11a080c262122bf1b59984e5e80f6095527bf507