NetApp fix for vsadmin role failure for ssc

The ssc feature does not work with vserver admin
role. If the user is a vsadmin then he does cannot
use the ssc feature fully. This fix checks the access
and issues apis appropriately and also handles error
conditions.

Closes-Bug: #1253657

Change-Id: Ibc4409e5d5e6e36bba41e5f59dd94c05b87b1cde
Make sure report_interval is less than service_down_time

If service_down_time is less than report_interval, services will
routinely be considered down, because they report in too rarely.
Add check for service_down_time vs report_interval, if report_interval
is larger than service_down_time, automatically change
service_down_time to be report_interval * 2.5

DocImpact

Closes bug #1255685

Change-Id: I9b291669ea201321b03a48d2a132810f3bace2dc
GlusterFS: Complete snapshot_delete when info doesn't exist

The snapshot_delete operation will fail if the snapshot info file
doesn't contain a record for the snapshot, or does not exist.
This happens in cases such as when snapshot_create fails to commit
anything to disk.

The driver should allow the manager to delete the snapshot
in this case, as there is no action required for the driver
to delete anything.

Closes-Bug: #1252864

Change-Id: I8686a1be09dbb7984072538bff6c026bb84eeb52
Update hacking to hacking>=0.8.0,<0.9

Update hacking version to match version specified in requirements repo.

Fixed the following issues, which the newer version checks for:
print "" -> print("")
self.assertEquals -> self.assertEqual
self.assertNotEquals -> self.assertNotEqual

Change-Id: Ic4b70fd8f565cda28e23fe6b1da0e278f949373c
Closes-Bug: #1256738
Closes-Bug: #1256737
Closes-Bug: #1257274
delete.start/delete.end notification for hostless

Third party GUIs may rely on notifications to track the
progress of volume creation and deletion.  In the case that
a volume is being deleted after a failed attempt to create
(the volume is listed in the database but is not actually
resident in a backing store) the path that is taken in
volume.api.delete() makes no notifications of deletion
occurring.

This patch adds a volume_utils.notify_about_volume_usage
call to the beginning and end of the delete with a
delete.start and delete.end respectively.  The notifications
serve as triggers for GUIs to refresh the state of the
volume.  This change makes the hostless delete path's
functionality more consistent with the other paths through
the delete code.

Change-Id: I091b9d277834b341105569d41a48ef5c1fc105ce
Closes-Bug: 1257053
All API controllers inherit from wsgi.Controller

Most of the Cinder Public API Controllers inherit from wsgi.Controller,
but still some Cinder Public APIs controllers directly inherit from
object. So it's necessary to unify all the Cinder Public API Controllers
inherit from wsgi.Controller.
1. For code cleanup.
2. When we try to do body valid check in api's put/post method later, we
can use the helper function is_valid_body instead of using try...catch
block.

Change-Id: I88812224e0b182939cfccce63982cb8f95792891
Closes-Bug: #1257198
Fixes case insensitive for resp body

Create metadata for a snapshot with key-value set, which key in uppercase
and lowercase(e.g.{"key": "v1", "KEY": "V1"), the server accept the
request and return the key-value set {"key": "v1", "KEY": "V1"}. But the
server just add one metadata because the server is not case sensitive.

The patch will modify the resp body with the one which the server added.

update_all has the same ploblem.

DocImpact
Change-Id: I684049412a4aa84f593e970c87157c74fffdfffe
Closes-Bug: #1255917
Merge "Fixes case insensitive for resp body"
Merge "All API controllers inherit from wsgi.Controller"
VMDK:Using host mount info for datastore selection

Currently the VMDK driver picks a datastore for volume creation based
on space utilization. The shadow VM corresponding to the volume is
migrated to a different host and datastore if the volume is attached
to an instance created in a host which cannot access the volume's
current datastore. To reduce these migrations, this fix selects a
datastore which is connected to maximum number of hosts, provided
there is enough space to accommodate the volume. Ties are broken
based on space utilization; datastore with low space utilization
is preferred.

Closes-Bug: #1250816
Change-Id: I5ecf22d782857fca0889799229465c49afe188a0
Fixes case insensitive for resp body

Create metadata for a volume with key-value set, which key in uppercase
and lowercase(e.g.{"key": "v1", "KEY": "V1"), the server accept the
request and return the key-value set {"key": "v1", "KEY": "V1"}. But the
server just add one metadata because the server is not case sensitive.

The patch will modify the resp body with the one which the server added.

update_all has the same ploblem.

Fixes errors on v2 unittest without difficulty.

DocImpact
Closes-Bug: #1258004

Change-Id: Ic337c0a351ac234493e1d73b86ba87520f32289a
Merge "Update hacking to hacking>=0.8.0,<0.9"
Merge "NetApp fix for vsadmin role failure for ssc"
Fix docstring for snapshot_metadata controller

Change-Id: I8227c79c35688da03f5b850daec3215d7c79b2a6
Closes-Bug: #1258000
Adds lock for create from vol/snap to avoid race conditions

This patch protects create from volume/snapshot by using a
lockfile to protect the operation from concurrent deletes of
the volume/snapshot used in the create operation.

Currently, if a volume/snapshot is deleted while a volume is
being created from it that delete may complete during the
create operation thus leaving the new volume in error or stuck
state. This lock will ensure that:

(a) if a create of VolA from snap/volB is in progress, any
    delete requests for snap/volB will wait until the create
    is complete.

(b) if a delete of snap/volA is in progress, any create from
    snap/volA will wait until snap/volA delete is complete.

Co-authored-by: Takashi Natsume <natsume.takashi@lab.ntt.co.jp>
Closes-Bug: 1251334
Change-Id: Ie4bc0af789ab232593f55aa2f6b34345eb9b9929
Catch ImageBusy exception when deleting rbd volume

If we try to delete an rbd volume that has 'watchers' on it
i.e. client connections that have not yet been closed
possibly because a client crashed, the remove() will throw an
ImageBusy exception. We now catch this exception and raise
VolumeIsBusy with a useful message.

If the volume delete fails in this way it will now stay as
'available' instead of going to 'error_deleting' so that the
delete can be retried (since it is expected to work on a
retry after waiting for the connection to timeout).

Change-Id: I5bc9a5f71bdb0f9c5d12b5577e68377e66561f5b
Closes-bug: 1256259
GlusterFS: Remove glusterfs_disk_util option

The glusterfs_disk_util option lets one choose whether to
use df or du to calculate space usage.  This serves no real
useful purpose.  Should remove this option to simplify things.

Since this does not meaningfully impact behavior, removal of
this option shouldn't require any special deprecation efforts.

(This same change was made for the NFS driver in 5bf7b9be.)

DocImpact: Remove config option

Change-Id: I302692454b43de15688d65873a72a3dada0c67cb
LVM: Create thin pools of adequate size

Thin pools in LVM are quite different from volume groups or logical
volumes and their differences must be taken into account when providing
thin LVM support in Cinder.

When you create a thin pool, LVM actually creates 4 block devices.  You
can see this after thin pool creation with the following command:

    $ dmsetup ls

    volumes--1-volumes--1--pool       (253:4)
    volumes--1-volumes--1--pool-tpool (253:3)
    volumes--1-volumes--1--pool_tdata (253:2)
    volumes--1-volumes--1--pool_tmeta (253:1)

In the above command, a thin pool named 'volumes-1-pool' was created in
the 'volumes-1' volume group.  Despite this, the 'lvs' command will only
show one logical volume for the thin pool, which can be misleading if
you aren't aware of how thin pools are implemented.

When you create a thin pool, you specify on the command line a size for
the pool.  LVM will interpret this size as the amount of space requested
to store data blocks only.  In order to allow volume sharing and
snapshots, some amount of metadata must be reserved in addition to the
data request.  This amount is calculated by LVM internally and varies
depending on volume size and chunksize.  This is why one cannot simply
allocate 100% of a volume group to a thin pool - there must be some
remaining space for metadata or you will not be able to create volumes
and snapshots that are pool-backed.

This patch allocates 95% of a volume group's free space to the thin
pool.  By doing this, we allow LVM to successfully allocate a region for
metadata.  Additionally, any free space remaining will by dynamically
used by either data or metadata if capacity should become scarce.

The 95/5 split seems like a sane default.  This split can easily (and
probably should) be made user-configurable in the future if the user
expects an abnormal amount of volume sharing.

Change-Id: Id461445780c1574db316ede0c0194736e71640d0
Closes-Bug: #1245909
LVM migrate: Use keywords for the brick instance

In the `migrate_volume` method a new instance is created when the
dest_vg is not equal to the source vg. This new brick instance is
created using positional arguments instead of keywords. However, some of
those arguments are passed in the wrong positions.

This patch uses keywords for the misplaced arguments.

The patch also changes `test_lvm_migrate_volume_proceed` in order to
fully test the happy path and catch things like this.

Closes-bug: #1258128
Change-Id: I75cf91171709554053fd5b52a4aae4e176e8364e
Convert lvm_mirrors to int

When the location_info is parsed and `lvm_mirrors` extracted, it is not
being converted to int. This causes some boolean checks to return True
even when there are not mirrors enabled because of the u'0'.

Closes-bug: #1258133
Change-Id: I67c463d9f139fe326bbcab4d193c45459129b514