Don't truncate osapi_glance_link or osapi_compute_link prefixes

When osapi_glance_link_prefix or osapi_compute_link_prefix is
defined and used to update the links return in API responses, do
not drop the path component of the overriding link prefix.

Change-Id: I290e3f85b94e28d25a54621c1699081cbbf48456
Closes-Bug: 1304099
VMware: Do not read opaque type for DVS network

Save backend calls when using DVS VIF type. There is no need to
get the opaque network in the event that we have a DVS network.

Change-Id: I75973005193f862dfde3b99167cb53d3114a8349
Adds verbosity to child cell update log messages

Child Cells frequently update their parents with their capabilities and
capacities. The debug log message contains the capabilities or capacities
that are being updated. This patch adds the names of the parent cells to
which the information is being sent to the debug log message.

Change-Id: Ic34191da191369e64a016c8300a8da294aaa9e07
Closes-Bug: 1351301
Remove pre-icehouse rpc client API compat

The compute rpc API was at version 3.23 for the Icehouse release.
Since we only support live upgrades from N-1 to N (Icehouse to Juno),
we can remove old rpc client support for versions that were
pre-Icehouse release.  This patch does that for the compute rpcapi.

Change-Id: Ice20b141dbe28de02e125fa5ced1dbc286ba2ee3
Fix wrong command for _rescan_multipath

The _run_multipath command is wrong in [1], it should be 'multipath -r'
not 'multipath - r'. And adds test for this.

[1]: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume.py#L590

Change-Id: I3c5d12cc11988d65a8fc9e6143d26c2d1220d253
Closes-bug: 1362916
Handle ExternalNetworkAttachForbidden exception

When creating an instance and assigning a public network
to it without admin authority, ExternalNetworkAttachForbidden
will be raised. But this exception is not handled in V3 api.

Change-Id: Id70d8eaa704fd82cef1afdb65756d75cb56fbdce
Closes-Bug: #1353949
Lock attach_volume

There are some issues with instance and volume cleanup when the volume
is not in a fully attached state so it will be safer to not attempt a
terminate_instance while there are attachments in progress.

Change-Id: I4347794e51004a881bf4ef5ee30f65ac28773e51
Closes-Bug: #1355348
Backport some v3 aggregate API unittest to v2 API

This patch add some v3 aggregate API unittest into V2 API to improve
V2 API unittesting.

Change-Id: Id4d533e74b462b55b155b4e4ad7d94e018d35726
libvirt: persist lxc attached volumes across reboots and power down

Currently, attached volumes to a LXC instance will be lost, when
hard rebooting or powering it down.
Adjusting the get_storage_config() to include LXC volumes in the
domain configuration.
The change will add all volumes to the instance config, a part from
the root device mapping.
When (re)creating a LXC instance, get_bdms_to_connect() generator
from block_device module, will be used to exclude the root device
mapping for a domain configuration.

Co-authored-by: Sam Morrison<sorrison@gmail.com>
Closes-Bug: #1269990
Change-Id: I0aa227ede5fca065c6da15506d8cc4391d993e20
libvirt: do not fail instance destroy, if mount_device is missing

In some cases, a volume does not have a mount_device field populated
in the device_mapping, especially for LXC instances. LXC instances
does not have a disk_bus mapping in get_dev_prefix_for_disk_bus()
and that results in mount_device being set to None.

Related to blueprint libvirt-start-lxc-from-block-devices

Change-Id: Idbd4ced0337c64003af08ac0497008878fb1c2e1
Fix EC2 instance type for a volume backed instance.

EC2 reports 'instance-store' type for a volume backed instance
which was booted with short root device name (no '/dev/' prefix).

This fix make instance type detection independent of addition
'dev' prefix to the instance root device name.

Change-Id: I6c2be09eaa91341e401bbc128b3efc79c60e345e
Closes-Bug: #1324400
VMware: Remove unused variable in test_configdrive

TrivialFix

Change-Id: I3b4c55f22e515d9a66311e4c608febf617babc5a
Fixes Hyper-V boot from volume root device issue

Checks that instances are booting from volume by checking
the correct root device obtained from the block device info.

Change-Id: I94941dfe96c1596c8e9b91df3d3d19d33ae7fe92
Co-Authored-By: Zsolt Dudas <zdudas@cloudbasesolutions.com>
Closes-Bug: #1357972
tests: fix test_compute to have predictable service list

The _create_service_entries() method in test_compute.py accepted
a dict for the list of services. The tests would then pull out
the first item in the list of keys of the dict. When the python
hash seed is set to a random value the list of keys has no
predictable ordering, causing the test suite to break. Changing
the data structure to use a list of lists instead of a dict
ensures strict ordering.

Related-bug: #1350287
Change-Id: I01a0d4cfb086eb94b49d25b65daf08e55e2087b4
Fix for volume detach error when use NFS as the cinder backend

For NFS volumes, every time you detach a volume, nova tries to umount
the device path. If the device path is busy (or in use), it should
output a message to log and continue.
In current code, if the device is busy, it cannot catch the ‘device is busy’
message returned by umount, because it looking for the ‘target is busy’.
 Therefore, current code skip the ‘if’ statement and run the ‘else’ and
raise the exception.
Fix: Add ‘device is busy’ to if statement.

Add a mock test to check the behaviour of the
virt.libvirt.volume.LibvirtNFSVolumeDriver.disconnect_volume
when it has umount errors.

Closes-Bug: #1340552

Change-Id: Iac946c37064c5f5bf5a102305de40d21d16846c1
Make floatingip-ip-delete atomic with neutron

The infra guys were noticing an issue where they were leaking floating ip
addresses. One of the reasons this would occur for them is they called
nova floating-ip-delete which first disassocates the floating-ip in neutron
and then deletes it. Because nova-api makes two calls to neutron if the
first one succeeds and the second fails it results in the instance no longer
being associated with the floatingip. They have retry logic but they base it on
the instance and when they go to retry cleaning up the instance the floatingip
is no longer on the instance so they never delete it.

This patch fixes this issue by creating a new method
disassocate_and_delete_floating_ip which contains the different logic
needed to atomically do this for neutron rather than putting the logic
in the api layer to do something differently if one is using neutron.

This patch also adds additional code coverage for nova-network in the
delete floatingip case.

Closes-bug: 1356157
Change-Id: I53b0c9d949404288e8687b304361a74b53d69ef3
Deprecate cinder_* configuration settings

Create a new section in the configuration file called 'cinder'.
Move all of the cinder_* configuration settings to this section.

DocImpact

The table below has the changes:

+---------------------------------+----------------------------+
| 'DEFAULT' Section               | 'cinder' Section           |
|---------------------------------|----------------------------|
| cinder_catalog_info             | catalog_info               |
| cinder_endpoint_template        | endpoint_template          |
| os_region_name                  | os_region_name             |
| cinder_ca_certificates_file     | ca_certificates_file       |
| cinder_http_retries             | http_retries               |
| cinder_http_timeout             | http_timeout               |
| cinder_api_insecure             | api_insecure               |
| cinder_cross_az_attach          | cross_az_attach            |
+---------------------------------+----------------------------+

UpgradeImpact
The changes are backward compatible.

Change-Id: Ie6b3f9d60b5e149375ff3039be444538f5f50756
Make strutils.mask_password more secure

Make some enhancements to strutils.mask_password to allow it to
catch more cases of passwords in strings.

The following is a listing of patterns that will be handled. The
keyword that mask_password uses (a list of four now) is represented
by <key> and the password is shown as <password>. Quotes (both single
and double) are represented as <quote>.

--<key> <password>
--<key> <quote><password><quote>
<key> = <password>
<key> = <quote><password><quote>

All existing tests and patterns are still handled.

Cherry-pick from review.openstack.org/113407
Partial-Bug: #1345233

Change-Id: I693baaf225c5b393401d499d14f271cadb63eb2c
Mask passwords in exceptions and error messages

When a ProcessExecutionError is thrown by processutils.execute(), the
exception may contain information such as password. Upstream
applications that just log the message (as several appear to do) could
inadvertently expose these passwords to a user with read access to the
log files. It is therefore considered prudent to invoke
strutils.mask_password() on the command, stdout and stderr in the
exception.

Cherry-pick from review.openstack.org/109417
Partial-Bug: #1343604

Change-Id: I9741dcdebdb4bed295ddc5ec4c4b117fffcfe88c
Fix instance boot when Ceph is used for ephemeral storage

is_block_dev attribute was mistakenly set to True for RBD images
(we don't actually map RBD images to block devices in the *host*
system, but only in *guests*, so as far as the host system is
concerned an RBD image must not be treated like a system block
device). This led to a situation when mkfs was called for a non
existing file and failed when trying to create an ephemeral fs.

Closes-Bug: #1362221

Change-Id: I54c0d117a53bb61f278b2e137bd29a595548f5a1