NetApp direct to filer drivers for iscsi and nfs.

The drivers are designed to work with 7 mode and cluster storage systems.
The drivers are designed to communicate with the NetApp storage controllers
directly without requiring any intermediate management softwares in place using
the NetApp api. Hence these represent the direct to filer versions of NetApp drivers.
This also contains conventional nfs driver for cluster systems which requires
NetApp management software as middle layer. Implementation for cloning support
and basic volume type scheduler support in driver added. All NetApp drivers
shifted into separate NetApp package. NetApp api refactored into a separate file
other than drivers.

blueprint netapp-direct-volume-drivers
blueprint netapp-cluster-nfs-driver

Change-Id: If3f5389d1f98eba899c4928a8ae1c1d3430bd847
make test_databases instance variable

eventually we'll need to do this to allow for testr testing, but in
doing so this also exposed that sqlite downgrade of source_volid
didn't work when data was in the table (sqlite just happily ignored
the drop, then failed when trying to added the column for a second
time during the snake walk.)

This fix also adds the correct sqlite downgrade path for migration
005 to actually drop source_volid, but ensure we don't loose any
other data in the process.

Change-Id: I082b53e108c4d564e33ef79979ea8c1642afdbcd
add postgresql opportunistic testing

sync _reset_databases from nova, which is needed for working
postgresql. Add opportunistic testing for postgresql, which will
run in CI.

Change-Id: I8166c2f723cabc5c5c07c7babaec8ffed5cee650
v2 volume/snapshot create will correctly give a 202 response

Before it was giving a 200 response. Since the request was received fine
but still has to process things, this is a more correct response. In
addition this is more consistent with other OpenStack project create
rest api calls.

Change-Id: I4f29cda7638c257c9cb5be7a5edf2bc4c48626d8
change display_description to description in volumes

Rest v2 api volume responses key display_description changes to
description to be consistent with other modules and projects.

Change-Id: Ibafa2865c6c4a73fc1cf4694ae77a6709fd2e6e5
Update snapshot rest api to be consistent with volumes

display_name becomes name and display_description becomes description in
both parameters to pass in requests and keys returned from responses.
This does not update the snapshot model as it takes the same approach
with updating the volume rest api.

Change-Id: I413b5f87882a216b7a847fb12c31f5e5ed91510b
Merge "NetApp direct to filer drivers for iscsi and nfs."
Merge "v2 volume/snapshot create will correctly give a 202 response"
Merge "make test_databases instance variable"
Merge "add postgresql opportunistic testing"
Fix PEP8 violation

There was a PEP8 violation in cinder/tests/test_solidfire.py, thus
./run_tests.sh was failing. This patch fixes the error.

Change-Id: I444fbfa57550db5c75b80683054c0ca44f4ee657
rbd: update volume<->image copying

Use fetch_to_raw() to convert from any format when
copying an image.

Fix two bugs in copy_image_to_volume():
 1) use rbd format 2 if supported
 2) resize to match the expected volume size

Also implement copy_volume_to_image().

Change-Id: I5589666c33ead087876e850363ad869050120882
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Allow create_volume() to retry when exception happened

Due to the fact that certain volume back-ends cannot easily report
simple total_capacity_gb/free_capacity_gb for their internal
implementation complexity, scheduler is updated to let those back-ends
who report unclear capacity pass capacity filter, thus there is chance
create_volume() request would fail. In a more general case, when a
volume back-end failed to serve create_volume request for whatever reason
it'd be good that we have a mechanism to 'retry' the request.

So the idea is when volume manager catches the exception from
driver.create_volume() call, it checks if the request is allowed to be
rescheduled (requests that are not: clone volume and create volume from
snapshot while 'snapshot_same_host' option is true), it composes a new
request back to scheduler with additional information to mark this specific
back-end has been tried (so that scheduler may choose to skip this back-end
if needed).  Scheduler is (filter scheduler only, simple and chance
scheduler is not supported) is updated as well so that it only retry
scheduler_max_attempts times.  In order to skip/rule out previously tried
back-ends in next schedule task, a new RetryFilter is added.

Changes:
1) volume RPC API create_volume() is updated with new parameters
to save original request information in case rescheduling is needed.
This bumps volume RPC API to 1.4.

2) implementation of create_volume() method in volume API is
refactored in order to distinguish if a request is allowed to
do reschedule (i.e. currently create volume from source volume
bypasses scheduler, not rescheduling is allowed).

3) add reschedule functionality in create_volume() of volume
manager so that it's able to send the request back to scheduler.

4) add schedule_max_attempts config option in scheduler/driver.py

5) add RetryFitler

6) change scheduler_driver default option to FilterScheduler

Change-Id: Ia46b5eb4dc033d73734b6aea82ada34ba5731075
Add an ID to temporary volume snapshot object

The create_cloned_volum method creates a temp snapshot and
when it's done it uses the standard snapshot_delete method
in the LVM driver.

A change was added that logs a statement using the object[id],
but we weren't actually setting this on our temp object so we
would get a trace fo key DNE.  This change adds an id key to
our temp object.

Fixes bug: 1125457

Change-Id: I4d886f2738a3c394a3a3a6373045669cf6bf8aaf
Fix ordering of function args

Previous change swapped context and type-id params in function
declaration.  Put it back.

Also, no reason for storing the qos dict in the parameters, if
it's stored then we continuously have to update it, remove that
for now and come up with a cleaner method later.

Change-Id: I0820a48660bdfd0481be3681e33d7e2ff3246bb1
Create a RemoteFsDriver class

This class holds common functions for drivers similar to NFS.

Change-Id: I83d89c971fd0153df7702e2803bed66df1a759f6
Add GlusterFS volume driver

This driver enables use of GlusterFS in a similar fashion
as the NFS driver.  It supports basic volume operations,
and like NFS, does not support snapshot/clone.

To enable, set volume_driver to
 cinder.volume.drivers.glusterfs.GlusterfsDriver

Note that this requires a Nova libvirt GlusterFS driver
as well.

Adds config options: glusterfs_shares_config,
glusterfs_mount_point_base, glusterfs_disk_util, and
glusterfs_sparsed_volumes.

DocImpact

Change-Id: I3dd4018f0cb4db48348728ca66bae7918309bb32
Add LIO iSCSI backend support using python-rtslib

This patch enables LIO as an iSCSI backend for Cinder, using
python-rtslib.

To enable, set "iscsi_helper = lioadm" in cinder.conf.

This requires python-rtslib 2.1.fb27, which is available from pip.

Implements blueprint lio-iscsi-support
DocImpact

Change-Id: Ifb23de65f26a40997afd6148a1d0be39bcc8d196
Update EMC SMI-S Driver

1. Merged with Avishay's changes(https://review.openstack.org/#/c/
19808/3) and removed copy volume<->image from my driver.
2. Also did some refactoring based on Mike's comments
last time (See https://review.openstack.org/#/c/19979/).
Made the following changes in emc_smis_common:
- Added a member variable conn to save get_ecom_connection
and moved exceptions inside the helper function get_ecom_connection.
- Also moved exceptions inside find_pool and get_storage_type.
- Added test cases so get_ecom_connection, find_pool, and
get_storage_type will be covered.
3. Made changes in emc_smis_common to handle both iscsi and fc.

Change-Id: I56db3ba41215489f6afe840ce5310a31f28bf5ae
Add an update option to run_tests.sh

This commit adds an update option to run_tests.sh. This option
is used to rerun install_venv.py on an already installed venv.
This will then just update out of date packages with pip.

Change-Id: I96dc6940f536feb810d57b3316e9d712aa750d93