Use block_device_info at post_live_migration_at_destination

post_live_migration_at_destination make libvirt.xml to destination
host. when execute (live) block migration for vm made by bootable
volume, method post_live_migration_at_destination write a
libvirt.xml in destination host.
but it missed block disk information so moved libvirt.xml always
has a wrong disk information.
I add block_device_info in blockinfo.get_disk_info at
post_live_migration_at_destination.
This change will make libvirt.xml with valid block disk information.

Closes-Bug: 1271780
(cherry picked from commit 0486647e1c85947bc48cbdd279b3f49824a6e692)

Conflicts:
	nova/tests/virt/libvirt/test_libvirt.py
	nova/virt/libvirt/driver.py

Change-Id: I31b3f2ef5f03f9ef61524bdb7ae089b18e4cd254
Further harden boto version checking in EC2 tests

Use pkg_resources to determine correct boto interface
to use in API tests.  stable/havana version.

Fixes Bug #1239220

Rebased-By: Dirk Mueller <dirk@dmllr.de>

Change-Id: I455ce5f2aabfad7178be361844c6b9b7b0d9c946
handle empty network info in instance cache

The nw_info value [] was being incorrectly equated with None. This caused
infinite recursion when api._get_instance_nw_info returned [] and was wrapped
with @refresh_cache (e.g., _get_instance_nw_info in
nova.network.neutronv2.api.API).

Fixes bug #1251792.

Change-Id: I3034fbb20e790f2d6f22c50b74a9f9dcdf9081aa
(cherry picked from commit 8bc974be50bea6ff6beacda890d9a33fa90059b3)
Do not hide exception in update_instance_cache_with_nw_info

From time to time an exception is raised in this method causing
the nw_info cache not to be saved. If this occurs we should raise
as this error will cause later errors to occur. For example, one
won't be able to associate a floatingip with the instance as there
is no nw_info found in this table. In addition, the fixed_ips on
the instance won't be returned via the api.

This patch also stubs out update_instance_cache_with_nw_info in a
few tests where an exception was being raised previously but went
unnoticed as it was not reraised but now is.

Related-Bug: #1252849
Related-Bug: #1249065

Change-Id: Ic860f72210ba736e11c10df21c4cb7625e9c0928
(cherry picked from commit 4c03383f21bc13caf3fed4db5aa1317d37758d5c)
Persist image format to a file, to prevent attacks based on changing it

The attack is based on creating a raw image that looks like a qcow2
image, and taking advantage of the code that used 'qemu-img info' to
autodetect the image format.

Now we store the image format to a 'disk.info' file, for Qcow2 and Raw
images, and only autodetect for images that have never been written to
that file.

SecurityImpact

Conflicts:
	nova/virt/libvirt/imagebackend.py

Manual tweaks to some mocking in test_imagebackend.py

Change-Id: I2016efdb3f49a44ec4d677ac596eacc97871f30a
Co-authored-by: Nikola Dipanov <ndipanov@redhat.com>
Closes-bug: #1221190
Pass thru credentials to allow re-authentication

This is a backport of 4 fixes squashed into one because:

1. They need to all be merged together as they build on each
   other as problems were found in each change after it was
   merged on master.
2. The 3rd change won't pass Jenkins on it's own so it has
   to be squashed with the 4th and final change, so I'm just
   going to squash the entire topic branch together.

The change bugs fixed and cherry pick commit hashes are left
intact for each change.

Closes-Bug: #1241275
(cherry picked from commit 51e5f52e4cb60e266ccde71f205c91eb8c97b48b)

Cache Neutron Client for Admin Scenarios

Closes-Bug: #1250580
(cherry picked from commit 85332012dede96fa6729026c2a90594ea0502ac5)

Users with admin role in Nova should not re-auth with Neutron

Closes-Bug: 1250763
(cherry picked from commit 1c1371c78b990447aeaa4377b512f8887e6ff3ce)

Fix Neutron Authentication for Metadata Service

Closes-Bug:  1255577
(cherry picked from commit 652620d12f3afe6845e41d9762b52d23f44fd557)

============

Change-Id: I2858562b180f3e058a2da9d67bef02af80927177
plug_vif raise NotImplementedError instead of pass

The Hyper-V, VMware, Docker and PowerVM drivers raise
NotImplementedError from the plug_vifs and unplug_vifs
methods rather than pass.
The compute manager handles the NotImplementedError
case when initializing the host.

Fixed bug 1218593

(cherry picked from commit 6ef9262a6aa39b2da8023067386b832a35ecc45c)

Conflicts:

	nova/tests/virt/hyperv/test_hypervapi.py
	nova/tests/virt/vmwareapi/test_vmwareapi.py
Change-Id: Id17a1f691f46816eadfe257d9ac0cc46502318ef
VMware: fix exception when using multiple compute nodes

When there is more than one compute node running and one of the
nodes restarts it terminates on an exception that the resource is
not found.

The cause of the issue is that a vif plug was being attempted for
a resource that did not exist. The vif plug should have raised an
"NotImplemented" exception.

Change-Id: I5a3f1cc73a981173b6c2fa493de3aad10a7e97fd
Closes-bug: #1272286
(cherry picked from commit d46f4a5b6d17f20a9c1de9367af11c9f3e7a7ef0)
Merge "Use block_device_info at post_live_migration_at_destination" into stable/havana
Merge "Further harden boto version checking in EC2 tests" into stable/havana
Merge "Pass thru credentials to allow re-authentication" into stable/havana
User a more accurate max_delay for reconnects

In an HA deployment, a 60 seconds delay between reconnects can be quite
problematic. This patch changes the delay calculation by setting the max
delay to 5s and by changing the way it is increased.

Unfortunately, this is one of the places where both our main drivers are
not consistent. Rabbit's driver uses configuration parameters for this
whereas qpid's driver has never had one. However, I would prefer not
adding configuration paremeters to qpid's driver for the following
reasons:

    1. Most of OpenStack services depend on the messaging layer, hence
    they need it to be available. A 5s delay seems to be reasonable and
    I could argue the need of tune it further. Although so frequent
    reconnects can add load to the network, that wouldn't be the main
    issue if one of the brokers go down.
    2. We're trying to move away from configuration options towards using
    transport URL. This path is still not clear and I would
    prefer avoiding adding new options until we clear it out.

Closes-bug: #1281148

Change-Id: I537015f452eb770acba41fdedfe221628f52a920
(cherry picked from commit 8b628d1e024f787dbb93d508117d9148388c0590)
network_info cache should be cleared before being rescheduled

If an instance fails to boot due to a non-networking error the instance
then gets rescheduled and launched on another compute node. In these cases
deallocate_for_instance() is called which deletes the network ports
allocated though the info_cache for the instance is never cleared. This patch
adds a call to update_instance_cache_with_nw_info() which causes the cache
to get cleared out. Note: the cache is only cleared if the instance hasn't
been marked for deletion. This is due to how instance_info_cache_update()
is implemented.

Change-Id: If967884c9a6276f5949a7a04b597cedcce12ba09
Closes-bug: #1285158
(cherry picked from commit 3538a9cd06e1e46ec5698edf4018c8d3075b619b)
Fix 'icehouse-compat' processing for compute rpc

Commit 2c63cae921d024f39f7d14459873d20caa5cf9e0 added some havana
specific rpc handling to aid with the live upgrade from havana to
icehouse.  Unfortunately, the processing was broken for some
methods.  It would claim to be sending a '3.0' message, but would
be sending some of the arguments in an old format, causing some
methods to break when processed on the remote end.

This patch fixes up the logic to ensure that if messages are sent as
'3.0', that the format of the data sent in the message matches.

Note that this is a stable/havana specific change.

Change-Id: I54135effe4e083f73cb7d912ac9e82fd182e2b2d
Closes-bug: #1301436
VMware: bug fix for VM rescue when config drive is configured

The addition of the config drive feature broke the VM rescue. The
cause of the problem was setting the instance UUID to the name of
the rescue instance. This must be a UUID! The solution
is to pass the instance name to the spawn method instead of writing
this on the instance UUID.

Closes-bug: #1247427

Conflicts:

	nova/virt/vmwareapi/vmops.py

Change-Id: Ia8494b0c8099753b666bda18a19c4c9a3f764616
(cherry picked from commit 88861df2b95a9c66a1a9d1969ad9e560786bb848)
Add RBAC policy for ec2 API security groups calls

The revoke_security_group_ingress, revoke_security_group_ingress, and
delete_security_group calls in the ec2 API were not restricted by policy
checks.  This prevented a deployer from restricting their usage via
roles or other checks.  Checks have been added for these calls.

Based on commit d4056f8723cc6cefb28ff6e5a7c0df5ea77f82ef but modified
for the backport.

Closes-Bug: #1290537
Change-Id: I4bf681bedd68ed2216b429d34db735823e0a6189
Merge "plug_vif raise NotImplementedError instead of pass" into stable/havana
Merge "VMware: fix exception when using multiple compute nodes" into stable/havana
Update ensure()/reconnect() to catch MessagingError

The error handling code that gets connections reset if necessary
caught ConnectionError. It really needs to catch MessagingError,
which ConnectionError inherits from. There are other types of
MessagingErrors that may occur, such as InternalError, and they need
to cause the connection to reset, as well.

This fix has already been merged into oslo.messaging.

Closes-bug: #1303890
Change-Id: Ic5082b74a362ded8b35cbc75cf178fe6e0db62d0
(cherry picked from commit 234f64d608266f43d8856ff98c89ceba6699d752)
(cherry picked from commit fae88f6805f6a16a60c23da67ce06cc0c78a706d)
Merge "network_info cache should be cleared before being rescheduled" into stable/havana