Fix broken solidfire create-snapshot

A previous change to enable secifying size for cloned volumes
created introduced a bug that breaks the ability to create-snapshots
when using the SolidFire driver
(CID: I5628c7fa922780d6b0601e2daa79310a61085edc).

The problem is that we use a shared method in the SF driver for both
create-clone and create-snapshot, the added change only considers the
clone case and does a get ref['size'] but in the case of snapshots
this needs to be ref['volume_size'].

Change-Id: I50603b3ac43f2c3c7e7811ec34de078a268519f7
Closes-Bug: 1234970
(cherry picked from commit 3cd27defb1c820ceadd8bbc6bfd5a405a7376cdb)
VMware: Re-create session for RetrievePropertiesEx

When RetrievePropertiesEx API is called after session timeout, the driver
does not re-create session automatically.

This is a regression caused when moving from RetrieveProperties to
RetrievePropertiesEx API.

Fixes bug: 1235187

Change-Id: I8a1e2452dfb3029365d59eae7850d1a9363b25f0
(cherry picked from commit 2d4ea101252141486fa6fd61b100772fafd40102)
VMware: Disallow snapshot of attached volume

The current implementation perform snapshot of attached volume.
This is invalid because when we take snapshot of the backing, the instance
continues pointing to the readonly delta and tries writing to it.
This patch will disallow snapshot creation of attached volume.

We have similar issue with delete of snapshot, when deleted the instance will
point to an invalid disk file, since that will get merged with its parent.
Hence any writes by the instance will fail with disk not found error.
This patch will ignore delete snapshot of attached volume.

Since we disallow snapshot of attached volume, we disallow linked clone of
attached source volume and also upload of attached volume.

Fixes bug: 1236891

Change-Id: I6a91eac8529836d1fe1de73bf6b28ec098aa7c54
(cherry picked from commit 2451a027bee50c834890f9e049fbaf4bfa911546)
Support Huawei driver upgrade from grizzly to havana

To make the driver upgrade from grizzly to higher versions, the patch
checks the old host name firstly when creating and deleting host in
Huawei storage system.
If the old host name exists:
*get and use it directly when adding host.
*delete it directly when deleting host.

Closes-bug 1237189

Change-Id: I82a9dbc1ef0bb0b91771afea4aadec5afe65eecf
(cherry picked from commit 0c9dc5d0e01576969f09edc0c9c72a2aa91b248a)
long flashcopy operation may block volume service

Storwize family uses flashcopy for snapshot or volume clone. The
volume delete has to wait until flashcopy finishes or errors out.
The _delete_vdisk() will poll volume FlashCopy status in a loop.
This may block volume serivce heartheat since it is in the same
. The solution is to use openstack FixedIntervalLoopingCall
to run the FlashCopy status poll in a timer thread.

The cinder volume mananger will resume delete operation for those
volumes that are in the deleting state during volume service startup.
Since Storwize volume delete may wait for a long time, this can cause
volume service to have long delay before it becomes available.
A greenpool is used to offload those volume delete operations.

Change-Id: Ie01a441a327e1e318fa8da0040ae130731b7a686
Closes-Bug: #1203152
(cherry picked from commit 7aa4f65a8c17aa037deff0f5b534ed694c17e62a)
improves lvm version parsing for customised builds

supports_thin_provisioning now uses a regexp to ensure parsing of
lvm version succeeds when the build is customised; also adds a test
for a customised string parsing

Closes-Bug: #1237994
Change-Id: I49049a58bbdb5315b9d2d7c259a9324ca15d78cb
(cherry picked from commit 51fd5edb106e26d65696ce37a70eef6a4f75b1e2)
Merge "VMware: Re-create session for RetrievePropertiesEx" into milestone-proposed
Merge "Support Huawei driver upgrade from grizzly to havana" into milestone-proposed
Merge "improves lvm version parsing for customised builds" into milestone-proposed
Merge "VMware: Disallow snapshot of attached volume" into milestone-proposed
FK lookup failures during migration

There are a couple of cases where migrations have failed
upgrading from Grizzly to Havana, this seems to be isolated
to a couple of migrations so far and the error message is a
failure to lookup/associate the volume-id FK dependency.

It appears that this is caused by not setting the charset
in the migration, so the result is that the initial db setup
uses utf8 and the migrations are using the default latin1.

This patch goes through all of the migrations in Havana that
specify InnoDB and explicitly sets the charset to utf8 to match
the volumes table (and the other original tables).

Change-Id: I43b219ff5e4eea10a7391ad65ef68a80b7460370
Closes-Bug: #1233861
(cherry picked from commit 645a84f990c90e28548cf35b4b5f242eb0e0c286)
Fix lvm.extend_volume to pass Gig suffix

The extend function in the lvm driver was not converting
the cinder size value to Gigabytes before passing the call
to the vg module.  The result was that we would attempt to
extend a volume to "new size in Megabytes" which of course
is less than the current size since we do a Gigabyte string
conversion on create and everywhere else.

This change makes sure we pass the integer change through
the sizestr method to get the G suffix needed to work properly.

Change-Id: I070962a3aa7038f612e19a93ccaa60cbc13008f6
Closes-Bug: #1240287
(cherry picked from commit ec442e41d2d243003a42ed60ae862e96142a5cde)
Open stable/havana

Change-Id: I92db056e8237fe83a4930a2f138d169fbbf1315e
Fix Huawei drivers to support other host OSs

Huawei drivers create Linux hosts by default when attaching volumes.
This patch makes them also support Windows, XenServer, AIX, etc.
The default OS is still Linux if it is not specified.

Users need to configure the host OS types in Huawei XML configuration
file. They need to set the items like this:
<Host OSType="Windows" HostIP="10.10.0.1, 10.10.0.2, ..." />
<Host .../>

When attaching a volume, the driver will get the host IP from nova. We
compare that IP with the IP in "HostIP" to get the corresponding OS type.

Closes-bug: #1229759

(cherry picked from I36fd52b97f790f1c68eaf24b6c12e7ef5d16145d)

Change-Id: I6792aaae050b126088447a0df65e10c425b84102
NetApp fix for compression and dedup stats

This change fixes the incorrect stats reported
for compression and deduplication by NetApp
iscsi and nfs backend.

Change-Id: I3e97d953b3ea2231c3eb1f950b0ca6705b3b2286
Closes-Bug:#1236839
(cherry picked from commit 592d860481e04312448d9ae0338066bd475fad7e)
NetApp fix mirrored stats

This fixes the incorrect reporting of mirrorred
statistics for NetApp iscsi and nfs backend.
The uninitialized and broken snapmirror is
not reported and only snapmirrored state
is reported back.

Closes-Bug:#1242808

Change-Id: I4b9f788e1f99b78d12db23bd92b6a277d5f64cb3
(cherry picked from commit a9852d6e727316d9bc07caee3096c67dc98dbb7a)
Storwize: Fix iogrp availability check

Do not check if vdisk_count is greater than zero to determine iogrp
availability. This is wrong, and leads to the driver not being able to
use empty iogrps.

Closes-Bug: #1240395
Cherry-pick from https://review.openstack.org/#/c/52031/

Change-Id: I50456a0c618103f919ef0da7fb9e44b77a1d9bff
NetApp fix ssc volume filtering inconsistency

This fixes the situation when the actual netapp
volume data structure becomes inconsistent
because of operating on the actual object
than the copy of it. It also introduces some checks
before calculating volume stats.

Change-Id: I626f9ca9aa8c7cc41ba4aa6058f14cc040f7301f
Closes-Bug:#1243081
(cherry picked from commit 1d503937e841da27f5e3441775b2d208e44f5e75)
NetApp fix for 7mode iscsi volume stats

This fixes the bug for reporting correct
capacity information in case of 7mode
iscsi drivers.

Change-Id: Ie41009eca866830173809211d58470025be847e3
Closes-Bug: #1238967
(cherry picked from commit e4bde7adda4e885fe847e0bacc4d3fc84547e723)
Check if dir exists before calling listdir

Changes along the way to how we clean up and detach after
copying an image to a volume exposed a problem in the cleanup
of the brick/initiator routines.

The clean up in the initiator detach was doing a blind listdir
of /dev/disk/by-path, however due to detach and cleanup being
called upon completion of the image download to the volume if
there are no other devices mapped in this directory the directory
is removed.

The result was that even though the create and copy of the image
was succesful, the HostDriver code called os.lisdir on a directory
that doesn't exist any longer and raises an unhandled exception that
cause the taskflow mechanism to mark the volume as failed.

Change-Id: I488755c1a49a77f42efbb58a7a4eb6f4f084df07
Closes-bug: #1243980
(cherry picked from commit 1766a5acc5c948288b4cd81c62d0c1507c55f727)