Fixes 3par driver methods that were double locking

There were many cases where the 3par drivers(FC and iSCSI) were double
locking, e.g. we were locking in the individual driver and in ‘common’.
This patch set removes all the locks in ‘common’ except the one on
‘create_cloned_volume’.

The methods ’_get_volume_state’ and ‘_copy_volume’ are only called in
‘create_cloned_volume’, so those locks were removed as well. All the other
methods in ‘common’ are only called by the driver which provide the lock,
except ‘create_volume’ which is also called by ‘create_cloned_volume’.

Change-Id: I660fbd04824bd2837e06480c8298ea5a709fc82a
Fixes: bug 1175791
Fixes the 3PAR drivers CPG validation

The 3PAR drivers were not checking if the Common Provisioning
Group(CPG) exists in the virtual domain at volume create time when
the default was being overwritten by volume type extra_specs value.
It was also reporting the default CPG in the metadata returning
after creating the volume. Now it reporting the correct CPG and
setting the correct snapshot cpg if different from the default
or not set.

Fixes bug 1186427

Change-Id: Idef7a4f29854c82980789786f2414e5046c8d8fb
Added extra-spec key scoping to the 3PAR drivers

The Filter scheduler uses the extra-specs data to determine capabilities
and back-end, and it also enforces strict checking. As a result previous
method of setting all the 3PAR settings(cpg, snap_cpg, persona, provisioning)
on the 3PAR volume using extra-specs on a volume type will not work with the
default scheduler. The filter scheduler will look at these keys and fail to
schedule deployment of the volume because no suitable back-end reports those
capabilities.

This fix will allow the custom keys (ie: hp3par:cpg=TEST_CPG,
hp3par:provisioning=full,...) to be scoped, unfortunately this requires
additional checks in the common driver to parse the keys.

I'll update the 3PAR driver section of the OpenStack Block Storage Admin
Guide in a follow-up patch.
DocImpact
Fixes: bug 1194289

Change-Id: Id3fde3b3a185eedf2b8193fd0dc6324d904e8834
Update snapshot rest api to be consistent with volumes

display_name becomes name and display_description becomes description in
both parameters to pass in requests and keys returned from responses.
This does not update the snapshot model as it takes the same approach
with updating the volume rest api.

Change-Id: I413b5f87882a216b7a847fb12c31f5e5ed91510b
rbd: update volume<->image copying

Use fetch_to_raw() to convert from any format when
copying an image.

Fix two bugs in copy_image_to_volume():
 1) use rbd format 2 if supported
 2) resize to match the expected volume size

Also implement copy_volume_to_image().

Change-Id: I5589666c33ead087876e850363ad869050120882
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Fixes the provisioning on selected volumes for NetApp 7 mode.

The current direct driver for 7 mode provisions openstack
volume on all NetApp 7 mode volumes mixing openstack volumes
with all non openstack volumes. This fix allows volume selection
by the user to keep openstack volumes separate from non opnestack
block storage entities.

bug 1132834

Change-Id: I77b78ce304e72482724b6ff7c52795c60992243e
Allow create_volume() to retry when exception happened

Due to the fact that certain volume back-ends cannot easily report
simple total_capacity_gb/free_capacity_gb for their internal
implementation complexity, scheduler is updated to let those back-ends
who report unclear capacity pass capacity filter, thus there is chance
create_volume() request would fail. In a more general case, when a
volume back-end failed to serve create_volume request for whatever reason
it'd be good that we have a mechanism to 'retry' the request.

So the idea is when volume manager catches the exception from
driver.create_volume() call, it checks if the request is allowed to be
rescheduled (requests that are not: clone volume and create volume from
snapshot while 'snapshot_same_host' option is true), it composes a new
request back to scheduler with additional information to mark this specific
back-end has been tried (so that scheduler may choose to skip this back-end
if needed).  Scheduler is (filter scheduler only, simple and chance
scheduler is not supported) is updated as well so that it only retry
scheduler_max_attempts times.  In order to skip/rule out previously tried
back-ends in next schedule task, a new RetryFilter is added.

Changes:
1) volume RPC API create_volume() is updated with new parameters
to save original request information in case rescheduling is needed.
This bumps volume RPC API to 1.4.

2) implementation of create_volume() method in volume API is
refactored in order to distinguish if a request is allowed to
do reschedule (i.e. currently create volume from source volume
bypasses scheduler, not rescheduling is allowed).

3) add reschedule functionality in create_volume() of volume
manager so that it's able to send the request back to scheduler.

4) add schedule_max_attempts config option in scheduler/driver.py

5) add RetryFitler

6) change scheduler_driver default option to FilterScheduler

Change-Id: Ia46b5eb4dc033d73734b6aea82ada34ba5731075
Add an ID to temporary volume snapshot object

The create_cloned_volum method creates a temp snapshot and
when it's done it uses the standard snapshot_delete method
in the LVM driver.

A change was added that logs a statement using the object[id],
but we weren't actually setting this on our temp object so we
would get a trace fo key DNE.  This change adds an id key to
our temp object.

Fixes bug: 1125457

Change-Id: I4d886f2738a3c394a3a3a6373045669cf6bf8aaf
Fix ordering of function args

Previous change swapped context and type-id params in function
declaration.  Put it back.

Also, no reason for storing the qos dict in the parameters, if
it's stored then we continuously have to update it, remove that
for now and come up with a cleaner method later.

Change-Id: I0820a48660bdfd0481be3681e33d7e2ff3246bb1
Create a RemoteFsDriver class

This class holds common functions for drivers similar to NFS.

Change-Id: I83d89c971fd0153df7702e2803bed66df1a759f6
Add GlusterFS volume driver

This driver enables use of GlusterFS in a similar fashion
as the NFS driver.  It supports basic volume operations,
and like NFS, does not support snapshot/clone.

To enable, set volume_driver to
 cinder.volume.drivers.glusterfs.GlusterfsDriver

Note that this requires a Nova libvirt GlusterFS driver
as well.

Adds config options: glusterfs_shares_config,
glusterfs_mount_point_base, glusterfs_disk_util, and
glusterfs_sparsed_volumes.

DocImpact

Change-Id: I3dd4018f0cb4db48348728ca66bae7918309bb32
Add LIO iSCSI backend support using python-rtslib

This patch enables LIO as an iSCSI backend for Cinder, using
python-rtslib.

To enable, set "iscsi_helper = lioadm" in cinder.conf.

This requires python-rtslib 2.1.fb27, which is available from pip.

Implements blueprint lio-iscsi-support
DocImpact

Change-Id: Ifb23de65f26a40997afd6148a1d0be39bcc8d196
Update EMC SMI-S Driver

1. Merged with Avishay's changes(https://review.openstack.org/#/c/
19808/3) and removed copy volume<->image from my driver.
2. Also did some refactoring based on Mike's comments
last time (See https://review.openstack.org/#/c/19979/).
Made the following changes in emc_smis_common:
- Added a member variable conn to save get_ecom_connection
and moved exceptions inside the helper function get_ecom_connection.
- Also moved exceptions inside find_pool and get_storage_type.
- Added test cases so get_ecom_connection, find_pool, and
get_storage_type will be covered.
3. Made changes in emc_smis_common to handle both iscsi and fc.

Change-Id: I56db3ba41215489f6afe840ce5310a31f28bf5ae
Add an update option to run_tests.sh

This commit adds an update option to run_tests.sh. This option
is used to rerun install_venv.py on an already installed venv.
This will then just update out of date packages with pip.

Change-Id: I96dc6940f536feb810d57b3316e9d712aa750d93
Adding support for Coraid AoE SANs Appliances.

This driver provide support for Coraid hardware storage appliances
using AoE (ATA Over Ethernet) protocol.

Implements blueprint coraid-volume-driver

Reference to Nova patch libvirt-aoe :
https://review.openstack.org/21101

The following operations are supported :
-- Volume Creation with Volume Types
-- Volume Deletion
-- Volume Attach
-- Volume Detach
-- Snapshot Creation
-- Snapshot Deletion
-- Create Volume from Snapshot
-- Volume Stats

The driver only work when operating on EtherCloud ESM,
Coraid VSX and Coraid SRX Appliances.

Change-Id: I7c8dde0c99698b52c151a4db0fb1bb94d516db61
Merge "Allow create_volume() to retry when exception happened"
Merge "Adding support for Coraid AoE SANs Appliances."
Merge "Create a RemoteFsDriver class"
Merge "Add an update option to run_tests.sh"
NetApp bug fix for multibackend scenario.

The multibackend scenario using filter scheduler
did not work with NetApp drivers because of use
of FLAGS. Changed to use configuration to support
the multibackend scenario.

bug 1132637

Change-Id: I8ac5bd6d4269f703c3b3b0aa062c92f4412c25af