Brick connector revised fix for NFS drivers

This change fixes the error that comes
while getting brick connector and attaching volumes
in case of NFS drivers in cinder. The attribute for
mount point base was not passed to attach_volume method
as the connector initialization logic is common for all
types of protocols. It is fixed by populating
the required parameter in the RemoteFsConnector
by extracting it from connection properties. The infrastructure
for fixing this situation for remote fs drivers is implemented
in this fix. The drivers need to override the method
_get_mount_point_base in their implementation. NFS driver
implements it. GlusterFs implementation of the method to follow
in a separate patch. Hence marking it partial fix.

Change-Id: Id61a437fca1870529fa8b85f7fc2f73eae665294
Partial-Bug: #1238085
Enable object caching in cinder REST API requests

Allow the core API to cache resources, such as DB results, so that
extensions can use data already retrieved within the same API request
eliminating additional expensive DB calls.

Loosely based on commit 9f9fbc54e7336da10fc3056bdaca2ec7d01c7f94 from
nova.

Change-Id: If9f49faf7305287c0489ad6209cf19b8bec612cc
Partial-Bug: #1197612
Use cached volumes in REST API extensions

This patch caches db volumes in the core volumes API so that they can be
used in the host and tenant attribute API extensions in order to lower
the response time by avoiding costly individual db queries for each
volume in the REST API contributions.

Change-Id: Ie0cbe929379df32fddbdce1953f13b6f26208bff
Partial-Bug: #1197612
Using HttpNfcLease to transfer vmdk files.

The current VMware driver supported only "sparse" and "preallocated"
vmware_disktype property set in a "vmdk" glance image. Both of these were just
copied over as *-flat.vmdk files into the vmfs or nfs file system of the
underlying datastore. This was used during copy_image_to_volume() api.
Unfortunately for a vsan datastore this work flow breaks since there is no
access to the flat vmdk file in the underlying datastore.

This patch introduces a new vmware_disktype for a glance image called
"streamOptimized". This is a format generated when a VM/vApp is exported using
the HttpNfc APIs. AS the name suggests this is a highly optimized format for
streaming in chunks and thus would result in much faster upload / download
speeds. The driver's copy_volume_to_image() implementation now always uploads
the vmdk contents using HttpNfc api so that the glance image ends up in the
"streamOptimized" disk type. Also the driver's copy_image_to_volume()
implementation now understands a "streamOptmized" disk type and uses HttpNfc to
import that vmdk into a backing VM.

Note that the same "streamOptmized" glance image format will also be supported
by VMware nova driver. This change is in a different patch -
https://review.openstack.org/#/c/53976/

Patch Set 4: Removing changes to requirements.txt that got in by mistake.
Patch Set 5: Fixing a small bug around progress updates.
Patch Set 6: Addressing comments from Avishay.

Fixes bug: 1229998

Change-Id: I6b55945cb61efded826e0bcf7e2a678ebbbbd9d3
Check only our VG name when testing if VG exists

Rather than calling "vgs" we should just call
"vgs <cinder-vg-name>" which returns only information about the VG
we are querying.

Also hardens test command string matching to error on unexpected
calls.

Change-Id: I25580e34b6589e161a51274f22dbb9ed922e43ed
GlusterFS: Synchronize operations that manipulate qcow2 data

Operations that modify qcow2 chains or the GlusterFS driver's
snapshot info file should use locking so that more than one thread
cannot run these operations at the same time.

Without this it is possible for multiple threads to concurrently
modify snapshot information resulting in an incomplete snapshot
chain.

Closes-Bug: #1251425

Change-Id: I93c6ffecd9aec38560cb19085243390b9120b363
Add valid check and unit tests on quota class

1.Cinder quota class hard_limit’s value should larger than -1 (-1 is a flag
value for unlimited).
2.Add unit tests for quota class.

Closes-bug: #1245360

Change-Id: Ie483db1bdbd4fac875de26abab8ebebd96f4938b
Merge "Enable object caching in cinder REST API requests"
Merge "Use cached volumes in REST API extensions"
NetApp fix free space as zero during 1st vol stats update

NetApp clustered ontap drivers report space as zero
till first 60 seconds of driver start. This is causing
discomfort for some performance sensitive deployements.
This was due to async nature of the NetApp stats collection job.
Job is changed to sync at driver start to improve customer
experience.

Change-Id: I7d5cbf590897a0d328ece3a60516c92c0ad0ee7f
Closes-bug: #1253660
Merge "Check only our VG name when testing if VG exists"
Fixes inappropriate error message.

"Volume cannot be deleted while migrating" as error message in
create_snapshot is inappropriate while volume is migrating.

Change-Id: I6cfaf312f9b1edbf4e54f737664290d6ab3b6332
Closes-Bug: #1252175
Create snapshot throws 500 Internal Error

The server doesn't check whether the parameter "volume_id" is in request body.
So the 500 error has been thrown.

We should catch the KeyError and transfer the KeyError to 400(HTTPBadRequest)
instead of 500.

Change-Id: I8a1dde1fd6ed820b39995af434efacc2a27c9604
Closes-Bug: #1252179
Fix typo in cinder.volume.API.

'_valid_availabilty_zone' --> '_valid_availability_zone'

Change-Id: Ib79772039d663f0eb4fc8994cc8edc5154d75a02
Closes-Bug: #1252915
To fix test_get_dss_rp in test_vmware_vmdk.py

The test "test_get_dss_rp" in test_vmware_vmdk.py is not testing
the positive case as intended. Rather, it tests the negative case
(no datastores) which is already covered by
"test_get_dss_rp_without_datastores".

Change-Id: I5b4802e22218217e0338fc721fc0ce1efeaee4f1
Closes-Bug: #1251126
Add default quota class into DB during migration

For some time now use_default_quota_class has been the
default setting for Cinder.  Cinder, however, has not been putting
any defaults for the default quota class into the database.  This
resulted in any command that queried for the default quotas to cause
the message "Deprecated: Default quota for resource: <resource> is set
by the default quota flag: <quota flag>, it is now deprecated.  Please use
the default quota class for default quota."

This commit resolves this issue by setting the default value for volumes,
snapshots and gigabytes in the quota_class table at migration time if there
is not already a class_name of 'default' in the quota_classes table.

Unit tests are included with this commit.

Closes-bug 1233763
Change-Id: I457ed8a9b78492eda22e31dfc198b2ee051d3ece
LVM: Activate source LV before cloning from it

LVM may be configured to not automatically activate
thin-provisioned LVs.

Ensure they are activated when performing a clone, otherwise
dd will fail as the device does not exist in /dev/mapper/.

Closes-Bug: #1252423

Change-Id: Ibcb946ffe7804b1976bf1b1863c48340c8cc7fd5
Don't do glance v2 calls when config is set to v1

Fixes bug 1252549

Change-Id: I6d9509bfecd2f73040a0ad27aff8fc5f6fd0a899
Subclass vendor specific exceptions.

Up until now drivers have been able to define their
own custom exception classes.  In most cases these are
inherited from either a native python exception class
or at best the base cinder exception class.

The problem with this is that it makes it very difficult
for higher layers (such as volume manager) to do any intelligent
exception handling and the base Cinder.exception class is too broad.

This change takes the first step in cleaning this up.  We create
new exception classes for all drivers to inherit from:
  VolumeDriverException
  VolumeBackendAPIException
  BackupDriverException

This still allows the freedom to create custom exceptions for a
particular driver, however it also gives us a common exception
class to catch and check in higher levels.  Further refinement
and standardization will be helpful going forward but this
should give us a good start.

Change-Id: I8c2ca4eecf7a64a82846d6d24fc0239db086237e
Merge "LVM: Activate source LV before cloning from it"