Don't truncate osapi_glance_link or osapi_compute_link prefixes

When osapi_glance_link_prefix or osapi_compute_link_prefix is
defined and used to update the links return in API responses, do
not drop the path component of the overriding link prefix.

Change-Id: I290e3f85b94e28d25a54621c1699081cbbf48456
Closes-Bug: 1304099
Rollback quota when confirm resize concurrently completed

quota is reserved when confirm_resize is executed, if a concurrent
operation updated the status of migration, currently compute layer
will directly return back without rollback the quota and it can
lead to quota inconsistent issue layer.

Change-Id: I2217e9a4523d2783eccd8ef31c1fd65a604e2cc3
Closes-Bug: #1301690
Adds verbosity to child cell update log messages

Child Cells frequently update their parents with their capabilities and
capacities. The debug log message contains the capabilities or capacities
that are being updated. This patch adds the names of the parent cells to
which the information is being sent to the debug log message.

Change-Id: Ic34191da191369e64a016c8300a8da294aaa9e07
Closes-Bug: 1351301
Remove pre-icehouse rpc client API compat

The compute rpc API was at version 3.23 for the Icehouse release.
Since we only support live upgrades from N-1 to N (Icehouse to Juno),
we can remove old rpc client support for versions that were
pre-Icehouse release.  This patch does that for the compute rpcapi.

Change-Id: Ice20b141dbe28de02e125fa5ced1dbc286ba2ee3
Add decorator expected_errors for ips v3 extension

Add decorator expected_errors for ips v3 extesion, that's required
for all the v3 extension.

Change-Id: Iad071fd77bd3571087badc1d836275bd64d8dfc8
Fix wrong command for _rescan_multipath

The _run_multipath command is wrong in [1], it should be 'multipath -r'
not 'multipath - r'. And adds test for this.

[1]: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume.py#L590

Change-Id: I3c5d12cc11988d65a8fc9e6143d26c2d1220d253
Closes-bug: 1362916
Handle ExternalNetworkAttachForbidden exception

When creating an instance and assigning a public network
to it without admin authority, ExternalNetworkAttachForbidden
will be raised. But this exception is not handled in V3 api.

Change-Id: Id70d8eaa704fd82cef1afdb65756d75cb56fbdce
Closes-Bug: #1353949
Lock attach_volume

There are some issues with instance and volume cleanup when the volume
is not in a fully attached state so it will be safer to not attempt a
terminate_instance while there are attachments in progress.

Change-Id: I4347794e51004a881bf4ef5ee30f65ac28773e51
Closes-Bug: #1355348
libvirt: persist lxc attached volumes across reboots and power down

Currently, attached volumes to a LXC instance will be lost, when
hard rebooting or powering it down.
Adjusting the get_storage_config() to include LXC volumes in the
domain configuration.
The change will add all volumes to the instance config, a part from
the root device mapping.
When (re)creating a LXC instance, get_bdms_to_connect() generator
from block_device module, will be used to exclude the root device
mapping for a domain configuration.

Co-authored-by: Sam Morrison<sorrison@gmail.com>
Closes-Bug: #1269990
Change-Id: I0aa227ede5fca065c6da15506d8cc4391d993e20
libvirt: do not fail instance destroy, if mount_device is missing

In some cases, a volume does not have a mount_device field populated
in the device_mapping, especially for LXC instances. LXC instances
does not have a disk_bus mapping in get_dev_prefix_for_disk_bus()
and that results in mount_device being set to None.

Related to blueprint libvirt-start-lxc-from-block-devices

Change-Id: Idbd4ced0337c64003af08ac0497008878fb1c2e1
Fix EC2 instance type for a volume backed instance.

EC2 reports 'instance-store' type for a volume backed instance
which was booted with short root device name (no '/dev/' prefix).

This fix make instance type detection independent of addition
'dev' prefix to the instance root device name.

Change-Id: I6c2be09eaa91341e401bbc128b3efc79c60e345e
Closes-Bug: #1324400
VMware: Remove unused variable in test_configdrive

TrivialFix

Change-Id: I3b4c55f22e515d9a66311e4c608febf617babc5a
Fixes Hyper-V boot from volume root device issue

Checks that instances are booting from volume by checking
the correct root device obtained from the block device info.

Change-Id: I94941dfe96c1596c8e9b91df3d3d19d33ae7fe92
Co-Authored-By: Zsolt Dudas <zdudas@cloudbasesolutions.com>
Closes-Bug: #1357972
Add test for get_instance_disk_info to test_virt_drivers

Commit 2bed16c89356554a193a111d268a9587709ed2f7 added
get_instance_disk_info to the base ComputeDriver class
but didn't add a test for it in test_virt_drivers, so
it's added here.

This also fixes the wrong method signature in the fake
virt driver.

Note that this wouldn't have found the problem for the
hyperv or vmware drivers due to bug 1209421.

Change-Id: I5765720d8c3e0a7962b493fa49359514f3136544
Related-Bug: #1358719
Fix sample files miss for os-aggregates

There are some sample files for os-aggregates V3 API assigning
'deleted' element with '0', which should be assigned to 'false'.
This patch fixes it.

Change-Id: I6d7814ce98bc0c701d4eabefcbe64100345cb2d3
Closes-Bug: #1294556
Fix for volume detach error when use NFS as the cinder backend

For NFS volumes, every time you detach a volume, nova tries to umount
the device path. If the device path is busy (or in use), it should
output a message to log and continue.
In current code, if the device is busy, it cannot catch the ‘device is busy’
message returned by umount, because it looking for the ‘target is busy’.
 Therefore, current code skip the ‘if’ statement and run the ‘else’ and
raise the exception.
Fix: Add ‘device is busy’ to if statement.

Add a mock test to check the behaviour of the
virt.libvirt.volume.LibvirtNFSVolumeDriver.disconnect_volume
when it has umount errors.

Closes-Bug: #1340552

Change-Id: Iac946c37064c5f5bf5a102305de40d21d16846c1
Make floatingip-ip-delete atomic with neutron

The infra guys were noticing an issue where they were leaking floating ip
addresses. One of the reasons this would occur for them is they called
nova floating-ip-delete which first disassocates the floating-ip in neutron
and then deletes it. Because nova-api makes two calls to neutron if the
first one succeeds and the second fails it results in the instance no longer
being associated with the floatingip. They have retry logic but they base it on
the instance and when they go to retry cleaning up the instance the floatingip
is no longer on the instance so they never delete it.

This patch fixes this issue by creating a new method
disassocate_and_delete_floating_ip which contains the different logic
needed to atomically do this for neutron rather than putting the logic
in the api layer to do something differently if one is using neutron.

This patch also adds additional code coverage for nova-network in the
delete floatingip case.

Closes-bug: 1356157
Change-Id: I53b0c9d949404288e8687b304361a74b53d69ef3
Make strutils.mask_password more secure

Make some enhancements to strutils.mask_password to allow it to
catch more cases of passwords in strings.

The following is a listing of patterns that will be handled. The
keyword that mask_password uses (a list of four now) is represented
by <key> and the password is shown as <password>. Quotes (both single
and double) are represented as <quote>.

--<key> <password>
--<key> <quote><password><quote>
<key> = <password>
<key> = <quote><password><quote>

All existing tests and patterns are still handled.

Cherry-pick from review.openstack.org/113407
Partial-Bug: #1345233

Change-Id: I693baaf225c5b393401d499d14f271cadb63eb2c
Mask passwords in exceptions and error messages

When a ProcessExecutionError is thrown by processutils.execute(), the
exception may contain information such as password. Upstream
applications that just log the message (as several appear to do) could
inadvertently expose these passwords to a user with read access to the
log files. It is therefore considered prudent to invoke
strutils.mask_password() on the command, stdout and stderr in the
exception.

Cherry-pick from review.openstack.org/109417
Partial-Bug: #1343604

Change-Id: I9741dcdebdb4bed295ddc5ec4c4b117fffcfe88c
Fix instance boot when Ceph is used for ephemeral storage

is_block_dev attribute was mistakenly set to True for RBD images
(we don't actually map RBD images to block devices in the *host*
system, but only in *guests*, so as far as the host system is
concerned an RBD image must not be treated like a system block
device). This led to a situation when mkfs was called for a non
existing file and failed when trying to create an ephemeral fs.

Closes-Bug: #1362221

Change-Id: I54c0d117a53bb61f278b2e137bd29a595548f5a1