Implement import/export for SolidFire Driver

This adds the import/export (or manage) functioniality
to the SolidFire driver.

Partially Implements: blueprint add-export-import-volumes

Change-Id: I4e7d5fc7c66d8037f01efc47a4cf5e93d9dba662
Update ref used for notifications

When we update a snapshot status in the db, we need to
use the newly updated snapshot-ref object in the following
notification method call.  Otherwise we're sending a
notifciation message with the old/outdated status
information.

NOTE: This also fixes the unit test that was written to
pass with the bug that existed in the notification.

Closes-Bug: #1349805

Change-Id: I32b86d7edd836310c6da7adffa46e915361668f0
fix atom link in XML Version API

This patch fixes tag names where the tag name has ':'
for links templates.
Some examples are
'{http://www.w3.org/2005/Atom}link'
should be
['{http://www.w3.org/2005/Atom}link']

and

'test1:test2'
should be
['test1', 'test2']

Closes-Bug: #1311243
Change-Id: I6e7c068e41eb6b069c24ef9d6c46b726ea722074
Have task/flow logging go to a separate log file

To be able to point the taskflow logging at a task log
we need to be able to separate the logs coming from cinder
from the logs coming from taskflow (as well as the listener
that acts as the intermediary between both projects).

To start the process of having a detailed task log
location, add-in a new log setting that can be used for just
this purpose.

This is not the most ideal way of doing this, since it just
provides a sample logging configuration adjustment, but there
does not appear to be a better way (due to how
cinder/openstack/common/logging.py can't be modified or adjusted
with a better option).

Part of blueprint task-logging

Change-Id: I5e3946c71110ee1b923c2258f3a7eb83c358426b
EMC VNX Direct Driver Update for Juno

VNX Direct Driver has been contributed to Icehouse release.
This patch refactors driver and adds the following new features

* Array-based Backend Support
* FC Basic Support
* Target Port Selection for MPIO
* Initiator Auto Registration
* Storage Group Auto Deletion
* Multiple Authentication Type Support
* Storage-Assisted Volume Migration
* SP Toggle for HA
* Security File Support
* Advance LUN Features
    # Compression Support
    # Deduplication Support
    # FAST VP Support
    # FAST Cache Support
* Storage-assisted Retype
* External Volume Management
* Read-only Volume
* FC Auto Zoning

Certificate Test Results
    https://bugs.launchpad.net/cinder/+bug/1336640

CCLA SCHEDULE B SUBMISSION

Change-Id: Ib7edaefa5eceb8e8c01ec0ce0dcdada7eaa9dd08
Implements: blueprint emc-vnx-direct-driver-juno-update
VMware:Support for attaching disk to backing

This change adds support for attaching a virtual disk to an existing
backing VM. Currently volume creation from preallocated/sparse image doesn't
honour the disk type in the volume extra spec and adapter type in the image
meta-data. The workflow to address these issues requires the above mentioned
method. This change also sets the disk size of backing to at least 1MB which
is the minimum required by VIM APIs.

Closes-Bug: #1340315
Partial-Bug: #1284284
Partial-Bug: #1287185
Partial-Bug: #1287176

Change-Id: Icdb1f26d5c8eaa42408a14c57c7394d96dda078f
VMware:Disk type conversion during clone backing

This patch adds support for converting virtual disk provisioning
type during clone backing operation. Currently volume creation from
preallocated/sparse image doesn't honour the disk type property in
the volume extra spec. The workflow to address this issue requires
disk provisioning type conversion during clone.

Partial-Bug: #1287176
Partial-Bug: #1287185

Change-Id: Ie493768445b838d3e9447edb76d9845e972bc7c3
Enable checks for E711, E712 and E713

* E711 comparison to None should be 'if cond is None:'
* E712 comparison to True should be 'if cond is True:' or 'if cond:'
* E713 test for membership should be 'not in'

Change-Id: I1481209a85fa6db0343d0111e2b19d9fab4538a7
VMware: Volume from non-streamOptimized image

Volume creation from non-streamOptimized images (preallocated/thin/sparse)
has following problems:

1) Sparse vmdk image is not converted to appropriate virtual disk type
   suitable for attaching to a nova instance.
2) The adapter type in image meta-data is ignored while creating volumes.
3) The vmware:vmdk_type extra_spec property is ignored.
4) Virtual disk extent operation is called with a wrong parameter which
   might result in unwanted disk provisioning type conversion.

This patch fixes the first 3 problems using the following workflow:

a) Create a disk-less backing.
b) Create a virtual disk (single flat extent) from the non-streamOptimized
   image
c) Attach the virtual disk to the backing
d) Clone the backing (if needed) to perform disk provisioning type conversion

Closes-Bug: #1287176
Closes-Bug: #1287185
Closes-Bug: #1284284
Change-Id: Ib7e9fae81d69d2fe490a4b603337f3d5cee1138c
VMware: Update default task_poll_interval value

The task_poll_interval option is used as the polling interval of vCenter/ESX
task status. This patch updates the default value to 0.5s in order to reduce
the wait time for task completion. Similar change has been done in nova and
this change is needed to keep the driver config values consistent.

DocImpact
   Updated default value of task_poll_interval to 0.5 seconds.

Change-Id: I66b80c629e255d61047592455ca42632f93051b2
Closes-bug: #1278028
Change the exception type for Storwize/SVC driver

When create a volume from snapshot or clone a volume,
Storwize/SVC require the same size for the source volume
and target, if not, will raise VolumeDriverException.

For this case, InvalidInput is more appropriate.

Change-Id: I4383c2eb8758077b3029b862dd8b6f982e1850a7
Fix unit test test_import_record_with_verify

The test case test_import_record_with_verify is intended to call a
backup service class with a defined verify function. This verify
function wasn't overloaded by the fake service.

Change-Id: Ib5738e9c463856dd71a47717b7abb8dff785b3b1
Closes-bug: #1350699
Add retry_on_deadlock to db update methods

There's some known races that have been addressed in
Nova via a retry_on_deadlock deocrator.  Some known
cases in Cinder exist as well when dealing with updates
to quotas and reservations_expire.

This patch introduces the decorator to Cinder and hits
the methods known to be susceptible to deadlock as well
as some others that seem as though they'd fall into the
same category.

Leverages work that's already been done in Nova

Co-Authored-By: Vish Ishaya <vishvananda@gmail.com>

Change-Id: Ic807f4f8a333e4d5477f86e41eea40191637087c
Partial-bug: #1350466
Fixes wrong usage of mock.assert_not_called()

There is no method assert_not_called in mock, use
assertFalse(mock.called) instead of that.

Change-Id: I1b37c49e0a624de42d89890b44458ec6bd4239ee
Use auth_token from keystonemiddleware

auth_token middleware in python-keystoneclient is deprecated and has
been moved to the keystonemiddleware repo.

Closes-Bug: #1342274
Change-Id: I1aadbe24db63eb2507b088cd53886d7f2e192cab
Fix snapshot id for snapshot_destroy

The id passed to snapshot_destroy should be snapshot['id'],
not volume['id'].

CCLA SCHEDULE B SUBMISSION

Change-Id: I1d84f1d596ebf014c0e3ea4202902ff515b3172a
Closes-Bug: 1351997
RPC client lazy initialization

All *Command classes are instantiating for each cinder-manage command
execution. RPC client should be initialized for VolumeCommands usage.
Some custom RPC clients (oslo.messaging backends) could init RPC
connection after client initialization so it is no need to create RPC
client for every cinder-manage command invoke.

Related-Bug: #1348787
Change-Id: I7b3388dded6eeec972d9949538e1608c3a538734
Merge "Add retry_on_deadlock to db update methods"
Merge "Use auth_token from keystonemiddleware"
Merge "Change the exception type for Storwize/SVC driver"