Fix cinder error for deprecated Netapp drivers

We deprecated some NetApp drivers in
this release. This change fixes c-vol
breakdown in case of cinder configured for
deprecated drivers and helps
in upgrade scenarios between releases
by printing appropriate warning and
suggestion message for deprecated drivers.

bug 1217220

Change-Id: I533d42257d93f706303b475fb4f222adcc0df3ed
NetApp storage service feature support

This change introduces storage service feature for
NetApp drivers. It reports back supported service
features like mirroring, deduplication, compression
etc. to the scheduler which can be used to shortlist
backends. It accepts extra_specs defined
in volume type with the service features and provisions
volumes based on the requested spec.

blueprint cinder-nfs-driver-qos

Change-Id: I0706af2fe39b2db111a6d98b016307fc63f74dc6
NetApp NFS efficient clone_image impl

This change implements clone_image funcionality
for NFS drivers. It implements an internal image cache
in nfs shares which will be used to clone images in case
the image_location supplied does not represent an nfs share
which could be used to directly clone image. It also implements
an internal cache cleaning mechanism at regular interval.

blueprint netapp-cinder-nfs-image-cloning

Change-Id: I20eaddf9febc78df33e31bd41b4698b772da3ae7
Increase test coverage brick/initiator/connector

Covered next methods:
	disconnect_volume_multipath_iscsi;
	get_multipath_iqn;
	get_iscsi_devices;
	get_multipath_device_name;

Change-Id: I0eaffa6d1408446f1f13e56016642df5337015a2
Fix volume_glance_metadata deletion

If volume_glance_metadata does not exist for a certain volume,
it returns an exception and does not delete volume from db.
Avoid deleting it if it does not exist.

Bug: 1209367
Change-Id: Ia607ec5ecfc115c1a593f14236ee93158938a8da
flake8 H202 error in test_image_utils.py

H202 will detect following case:
self.assertRaises(Exception, foo)

Replace Exception with test.TestingException to pass the H202 checking.

Bug #1214365

Change-Id: Idbd13d11995e6a22e99bb9d63c344a5b1a75f987
Clean up the sqlalchemy migrate manage.py script.

This way we can run it without figuring out passing in
the correct url and repository information.

Change-Id: Ib0f2c9d5c3888a1c094c8b636d735832a6242b7e
Set bootable flag for volume cloned from image

In previous code we don't set the bootable flag for volume
cloned from image. This does not appear to break booting
from volume at present but the status displayed by the cinder
client is broken.

The bug is reported and fixed several months ago(bug #1185533)
but the author doesn't provide unit test with the patch. Now it
has been broken again by somebody else.The patch re-fixed this
bug together with unit tests.

Also add a function named _handle_bootable_volume_glance_meta,
it is a combination of enable_bootable_flag and glance_metadata
handling. There are 3 kinds of volume creation tasks may required
to copy/fetch the glance metadata. In previous code every kind of
task have its own handling code, this patch consolidate them together
to share some common code.

fixed bug #1211272
fixed bug #1185533

Change-Id: I86007e26efbebe58dc0c5995b6f14f68258144b5
emit warning while running flake8 without virtual env

run_tests.sh -N -p
it will call the flake8 installed on your host to detect PEP8, and
the flake8 plugin "OpenStack hacking" may not installed on your
host, so this command may not detect the OpenStack Style Commandment
supplied by hacking(e.g H202).

run_tests.sh -p
it will call the flake8 from virtual env, flake8 plugin "OpenStack
hacking" installed in virtual env will be triggered.

The result from "run_tests.sh -p" should be trusted, and jenkins uses
virtual env to run flake8 too.

When "-N" is enabled, emit warning to remind user.

Bug #1208584

Change-Id: Ie08e5fa2b44088dad135e11583c046650d65df1e
Re-enable a lot of cinder scheduler tests

This highlight and fixes a NoneType exception in the schedulers
_get_weighted_candidates method

This was originally added to get around the fact that the filter tests
required that cinder was correctly installed on the system. But the
is_cinder_installed condition always returning False and this hid
problems of old packages been installed in our system. Also
930f5891b0815e1b49b9b2cc840e0c24b2796e84 added the setup.py automatically
to run_tests.sh so we can remove this.

See https://review.openstack.org/#/c/20213/ for history.

Fixes bug: 1213226

Change-Id: I86fee802c0543355471ddbd712e52ccec750cea0
GPFS use clone_image for creating volumes

If both source and target of gpfs create volume from image operation
are backed by gpfs storage use clone_image method for implementing
the move of image data to the new volume.

The copy_image_to_volume method is used only if this is not true,
and uses image_utils.fetch_to_raw to move image data to the new
volume.

Fixes bug #1213248

Change-Id: I3958febe67cc86bc3cb608288f7d064f74d3a731
Remove unused methods from cinder.utils

These methods were left over from the split from Nova and are unused still.

Remove methods:
	fetchfile()
	trycmd()
	generate_uid()
	last_octet()
	get_my_linklocal()
	parse_mailmap()
	str_dict_replace()
	flatten_dict()

Change-Id: I5d41dec66246814bb1edb6f4993d5307587a00f9
3PAR driver add missing domain check on QOS cmd

The 3PAR drivers need to support CPGs that are not part of a
virtual domain on the 3PAR backend. This patch now checks
if the domain is set before calling the createvvset command.
This instance was overlooked in the following patch that has
already been merged: https://review.openstack.org/#/c/42697/

Also fixes: bug 1212328

Change-Id: I0375dfc89104775582b9258f18f97c683e6148b1
Dont crash service if sf cluster isnt available

Currently if the SolidFire driver is configured but the cluster
isn't available for some reason the ensure_export call will raise
an unhandled exception and crash the volume-service.

We should be able to handle things like loosing connectivity to
a single back-end without impacting the other volume-service
backends.

We'll wrap the ensure_export call in a try block here and
return None in the case that the connection can't be made.
This will keep the service from crashing and log an error
message that the connection timed out.

Additional work would include adding a periodic retry task
to the manager to try and start the backend service for us
on some regular interval incase the device comes back.

Fixes: bug 1215064

Change-Id: Ice3f517d220c40113074bb77adbb10d5e32abd0b
Move root task class to base file.

Start cleaning up the large create_volume file by
splitting up the base task and task naming helper
into its own file.

Change-Id: Ibff80dc8b10507d49374737b4760039f51d8e1ba
Move the frequently injection task to the base folder.

Key/value injection is a typical way to bootstrap a
dependency based system (as there needs to be an
initial set of values to depend on). Since this inject
task is frequently used to accomplish this we should
just move it to the base file.

Change-Id: Ib49e949f319bc18744fc95f8a02fab1eade0fb6a
Implement QoS support for volumes

This patch is to implement generic Quality-of-Service(QoS) support for volumes.
The goal is to add an interface so that cloud/Cinder admins can use to set
volume QoS, which can be enforced either in hypervisor or on Cinder back-end
or both. QoS specifications are added as a standalone (only visible to admin)
entity.  So admin can create/update/delete and associate/disassociate QoS
specifications to other entities, in this case volume types.

Note that while it's possible for Cinder to set the granularity of QoS control
to every single volume, this patch puts the control granularity to the level
of volumes of the same type to minimize the impact of other Cinder parts.
In other words, the design is to bond QoS with volume types. So Cinder admin
can associate volume types with QoS specifications, and volumes of same volume
type share the same QoS specifications.

QoS can mean a lot different things that it's unlikely we can come up with a
interpretation that all vendors can agree on.  So the approach this
implementation takes is to make Quality-of-Service specs as free-from, i.e.
expressed as key/value pairs.

Changes:
 - Add a quality_of_service_specs table, using adjacency list relation to store
 a specs entry and its detailed specs in key/values. Note that to be able to
 distinguish where should the QoS specs be consumed, each QoS specs entity
 will have a 'consumer' (i.e. fixed key) with the value of where admin would
 like the QoS policy to be enforced/consumed, currently these three values are
 considered valid: 'front-end' (Nova Compute), 'back-end' (Cinder back-end),
 'both'. The default value for 'consumer' is 'back-end';
 - Add a new API extension 'qos_specs_manage' to allow list/create/update/
 delete/associate/disassociate of QoS specs;
 - Add volume/qos_specs internal API for qos specs manipulation;
 - Add 'qos_specs' info to data structure when
 initialize_connection() is called.
 - Add 'qos_specs' to request_specs and filter properties for
 a volume create request.

 TODO
 - Modify 'type_manage' API extension to be able to accept qos info.
 - Modify volume_types.create() to accept qos info and do the checks.

DocImpact

implement blueprint: pass-ratelimit-info-to-nova

Change-Id: Iabc61b941aaff10395b30e2045e3421369a317e2
Storwize/SVC: allow setting of I/O group

Allow setting the I/O group via config option or volume type. Also
updated comments and checks regarding iSCSI multipath, which is in Nova
and is not controlled by the driver.

Fixes: bug 1215363

Change-Id: Ia67fcb9991e21990cc81830145b8fd934a23488e
Merge "Dont crash service if sf cluster isnt available"
Merge "3PAR driver add missing domain check on QOS cmd"