Make sure report_interval is less than service_down_time

If service_down_time is less than report_interval, services will
routinely be considered down, because they report in too rarely.
Add check for service_down_time vs report_interval, if report_interval
is larger than service_down_time, automatically change
service_down_time to be report_interval * 2.5

DocImpact

Closes bug #1255685

Change-Id: I9b291669ea201321b03a48d2a132810f3bace2dc
GlusterFS: Complete snapshot_delete when info doesn't exist

The snapshot_delete operation will fail if the snapshot info file
doesn't contain a record for the snapshot, or does not exist.
This happens in cases such as when snapshot_create fails to commit
anything to disk.

The driver should allow the manager to delete the snapshot
in this case, as there is no action required for the driver
to delete anything.

Closes-Bug: #1252864

Change-Id: I8686a1be09dbb7984072538bff6c026bb84eeb52
delete.start/delete.end notification for hostless

Third party GUIs may rely on notifications to track the
progress of volume creation and deletion.  In the case that
a volume is being deleted after a failed attempt to create
(the volume is listed in the database but is not actually
resident in a backing store) the path that is taken in
volume.api.delete() makes no notifications of deletion
occurring.

This patch adds a volume_utils.notify_about_volume_usage
call to the beginning and end of the delete with a
delete.start and delete.end respectively.  The notifications
serve as triggers for GUIs to refresh the state of the
volume.  This change makes the hostless delete path's
functionality more consistent with the other paths through
the delete code.

Change-Id: I091b9d277834b341105569d41a48ef5c1fc105ce
Closes-Bug: 1257053
VMDK:Using host mount info for datastore selection

Currently the VMDK driver picks a datastore for volume creation based
on space utilization. The shadow VM corresponding to the volume is
migrated to a different host and datastore if the volume is attached
to an instance created in a host which cannot access the volume's
current datastore. To reduce these migrations, this fix selects a
datastore which is connected to maximum number of hosts, provided
there is enough space to accommodate the volume. Ties are broken
based on space utilization; datastore with low space utilization
is preferred.

Closes-Bug: #1250816
Change-Id: I5ecf22d782857fca0889799229465c49afe188a0
Fixes case insensitive for resp body

Create metadata for a volume with key-value set, which key in uppercase
and lowercase(e.g.{"key": "v1", "KEY": "V1"), the server accept the
request and return the key-value set {"key": "v1", "KEY": "V1"}. But the
server just add one metadata because the server is not case sensitive.

The patch will modify the resp body with the one which the server added.

update_all has the same ploblem.

Fixes errors on v2 unittest without difficulty.

DocImpact
Closes-Bug: #1258004

Change-Id: Ic337c0a351ac234493e1d73b86ba87520f32289a
Adds lock for create from vol/snap to avoid race conditions

This patch protects create from volume/snapshot by using a
lockfile to protect the operation from concurrent deletes of
the volume/snapshot used in the create operation.

Currently, if a volume/snapshot is deleted while a volume is
being created from it that delete may complete during the
create operation thus leaving the new volume in error or stuck
state. This lock will ensure that:

(a) if a create of VolA from snap/volB is in progress, any
    delete requests for snap/volB will wait until the create
    is complete.

(b) if a delete of snap/volA is in progress, any create from
    snap/volA will wait until snap/volA delete is complete.

Co-authored-by: Takashi Natsume <natsume.takashi@lab.ntt.co.jp>
Closes-Bug: 1251334
Change-Id: Ie4bc0af789ab232593f55aa2f6b34345eb9b9929
Catch ImageBusy exception when deleting rbd volume

If we try to delete an rbd volume that has 'watchers' on it
i.e. client connections that have not yet been closed
possibly because a client crashed, the remove() will throw an
ImageBusy exception. We now catch this exception and raise
VolumeIsBusy with a useful message.

If the volume delete fails in this way it will now stay as
'available' instead of going to 'error_deleting' so that the
delete can be retried (since it is expected to work on a
retry after waiting for the connection to timeout).

Change-Id: I5bc9a5f71bdb0f9c5d12b5577e68377e66561f5b
Closes-bug: 1256259
GlusterFS: Remove glusterfs_disk_util option

The glusterfs_disk_util option lets one choose whether to
use df or du to calculate space usage.  This serves no real
useful purpose.  Should remove this option to simplify things.

Since this does not meaningfully impact behavior, removal of
this option shouldn't require any special deprecation efforts.

(This same change was made for the NFS driver in 5bf7b9be.)

DocImpact: Remove config option

Change-Id: I302692454b43de15688d65873a72a3dada0c67cb
LVM: Create thin pools of adequate size

Thin pools in LVM are quite different from volume groups or logical
volumes and their differences must be taken into account when providing
thin LVM support in Cinder.

When you create a thin pool, LVM actually creates 4 block devices.  You
can see this after thin pool creation with the following command:

    $ dmsetup ls

    volumes--1-volumes--1--pool       (253:4)
    volumes--1-volumes--1--pool-tpool (253:3)
    volumes--1-volumes--1--pool_tdata (253:2)
    volumes--1-volumes--1--pool_tmeta (253:1)

In the above command, a thin pool named 'volumes-1-pool' was created in
the 'volumes-1' volume group.  Despite this, the 'lvs' command will only
show one logical volume for the thin pool, which can be misleading if
you aren't aware of how thin pools are implemented.

When you create a thin pool, you specify on the command line a size for
the pool.  LVM will interpret this size as the amount of space requested
to store data blocks only.  In order to allow volume sharing and
snapshots, some amount of metadata must be reserved in addition to the
data request.  This amount is calculated by LVM internally and varies
depending on volume size and chunksize.  This is why one cannot simply
allocate 100% of a volume group to a thin pool - there must be some
remaining space for metadata or you will not be able to create volumes
and snapshots that are pool-backed.

This patch allocates 95% of a volume group's free space to the thin
pool.  By doing this, we allow LVM to successfully allocate a region for
metadata.  Additionally, any free space remaining will by dynamically
used by either data or metadata if capacity should become scarce.

The 95/5 split seems like a sane default.  This split can easily (and
probably should) be made user-configurable in the future if the user
expects an abnormal amount of volume sharing.

Change-Id: Id461445780c1574db316ede0c0194736e71640d0
Closes-Bug: #1245909
LVM migrate: Use keywords for the brick instance

In the `migrate_volume` method a new instance is created when the
dest_vg is not equal to the source vg. This new brick instance is
created using positional arguments instead of keywords. However, some of
those arguments are passed in the wrong positions.

This patch uses keywords for the misplaced arguments.

The patch also changes `test_lvm_migrate_volume_proceed` in order to
fully test the happy path and catch things like this.

Closes-bug: #1258128
Change-Id: I75cf91171709554053fd5b52a4aae4e176e8364e
Convert lvm_mirrors to int

When the location_info is parsed and `lvm_mirrors` extracted, it is not
being converted to int. This causes some boolean checks to return True
even when there are not mirrors enabled because of the u'0'.

Closes-bug: #1258133
Change-Id: I67c463d9f139fe326bbcab4d193c45459129b514
LVM migration: Check if name is equal to dest_vg

The existence check of the destination volume group is wrong. It
currently checks if there's a VG with a name equal to the source's vg
name instead of checking if it's equal to the dest_vg.

Closes-bug: #1258203
Change-Id: Ia5d4acb24b94c6aa832107c7eb4b6996985af97f
Brick LVM: Set C locale when gathering PV info

Some locales use commas as decimal separators, which means
that code breaks when someone expects to be able to use the
PV size as a float.

Change-Id: Idceb405e021ea8a0f44e19be43bad1307fbbb51c
Related-Bug: 1219963
Merge "Catch ImageBusy exception when deleting rbd volume"
Returns thin pool free space calculated from actual usage

This change adds a hidden method which calculates the thin
pool free space from the data_percent LVM options and reports it
accordingly in _update_volume_stats()

Change-Id: Id6a69644505ca10811db458ea90ed10f643054b2
Closes-Bug: 1249782
Sync rpc fix from oslo-incubator

Sync the following fixes from oslo-incubator:

ef406a2 Create a shared queue for QPID topic consumers
e227c0e Properly reconnect subscribing clients when QPID broker restarts

Change-Id: I934a6ea6e9eb510513640870750ea37e6f076df9
Closes-bug: #1251757
Closes-bug: #1257293
Merge "LVM: Create thin pools of adequate size"
Add RBD test for volume not existing during delete

For cases when the volume does not exist in the backend, but Cinder
still knows about it, we just return from the rbd driver and let Cinder
do its thing.

Change-Id: I6150756bdd48259ec8d550fa7a5dc7a9f882ca22
Merge "LVM migrate: Use keywords for the brick instance"
Merge "Convert lvm_mirrors to int"