NetApp fix ssc volume filtering inconsistency

This fixes the situation when the actual netapp
volume data structure becomes inconsistent
because of operating on the actual object
than the copy of it. It also introduces some checks
before calculating volume stats.

Change-Id: I626f9ca9aa8c7cc41ba4aa6058f14cc040f7301f
Closes-Bug:#1243081
(cherry picked from commit 1d503937e841da27f5e3441775b2d208e44f5e75)
NetApp fix for 7mode iscsi volume stats

This fixes the bug for reporting correct
capacity information in case of 7mode
iscsi drivers.

Change-Id: Ie41009eca866830173809211d58470025be847e3
Closes-Bug: #1238967
(cherry picked from commit e4bde7adda4e885fe847e0bacc4d3fc84547e723)
Check if dir exists before calling listdir

Changes along the way to how we clean up and detach after
copying an image to a volume exposed a problem in the cleanup
of the brick/initiator routines.

The clean up in the initiator detach was doing a blind listdir
of /dev/disk/by-path, however due to detach and cleanup being
called upon completion of the image download to the volume if
there are no other devices mapped in this directory the directory
is removed.

The result was that even though the create and copy of the image
was succesful, the HostDriver code called os.lisdir on a directory
that doesn't exist any longer and raises an unhandled exception that
cause the taskflow mechanism to mark the volume as failed.

Change-Id: I488755c1a49a77f42efbb58a7a4eb6f4f084df07
Closes-bug: #1243980
(cherry picked from commit 1766a5acc5c948288b4cd81c62d0c1507c55f727)
Use pipe between ceph backup diff export/import

We now use a piped transfer between the rbd export-diff
and import-diff for incremental backups/restores as
opposed to holding the entire diff in memory.

Change-Id: I33476d9b3934781413af5cd2867a11d825a5d78e
Fixes: bug 1244464
(cherry picked from commit d384d28e1c2c7c3ce0bd676fb1c2bdb8a2d98a03)
Allow spaces in quoted SSH command arguments

The check_ssh_injection() method was rejecting arguments with spaces
even when they were quoted, this was causing problems with some volume
driver commands such as commands for a storage pool with spaces in the
name.

Note that this backport also fixes a typo that has been fixed separately
in master with commit eb0f2e4dd538a79184efbb23d7e404147dfe877b .

Closes-Bug: #1244415
Change-Id: Ie4b809e1b39fdb752cf634e6d3c0a3924d8ac52b
(cherry picked from commit 2737c76cb2fb436f117a4f635aebca7a01691d88)
Merge "NetApp fix for 7mode iscsi volume stats" into stable/havana
Merge "NetApp fix ssc volume filtering inconsistency" into stable/havana
Don't zero out thin provisioned LV's on delete

Thin provisioned LV's don't need secure delete to protect
from data leakage.  Also, zeroing these out kinda defeats
the purpose of using thing provisioning.

This patch add a check for the lvm type and if it's thin simply
returns from the lvm.clear_volume() method.

Change-Id: Ie6764209018152565295291efc6fbba553698ae6
Closes-Bug: #1240299
(cherry picked from commit de4392314f9b0def8fab65679ec5668aec98fbee)
Fail when image is bigger than the volume

When creating a new volume from a qcow2 image stored in glance, we will
be given the physical size of the image instead of the virtual size.
Most drivers will convert that image to raw after downloading for
resizing it to the requested volume size afterwards.

If the virtual size of the image is bigger than the requested one, the
resulting volume might end up being unusable after the resize even
though the creation went good.

This patch will make the volume creation fail if the virtual size of
the image exceeds the one of the requested volume.

Closes-Bug: #1235358
Change-Id: I254cd9e3acf5d9eb7eb913b37d27a14e97568dec
(cherry picked from commit 6e287c0f2bb7d4994d50f1763f412277e4dac6f7)
Merge "Allow spaces in quoted SSH command arguments" into stable/havana
Merge "Check if dir exists before calling listdir" into stable/havana
Add call to retrieve image metadata for volumes in bulk

When using the GET volume details REST API call, the image metadata API
contribution is making an individual db call for each of the available
volumes. When the number of volumes is large the details call can take
several minutes.

This patch adds a call to the volume.API to retrieve image metadata in
bulk, very similar to the one used to retrieve individual volume image
metadata.

Change-Id: Ic3aa721016704c72b7564cc5ceff71676806a24a
Partial-Bug: #1197612
(cherry picked from commit bb2daca2864a5607635a3aa37f3874fdda22797b)
Retrieve volume image metadata using single query

The image metadata REST API contributions are making an individual db
call for each of the available volumes. When the number of volumes is
large the volume details call can take several minutes.

This patch changes the image metadata API contributions to take
advantage of the new db query to retrieve metadata in bulk.

Change-Id: I9a35438c1f38ea8a3d8f5b687ae58ba1f3f78121
Partial-Bug: #1197612
(cherry picked from commit 56e5805203a14acad78dc806247b5f0f4a22c9f8)
Enable object caching in cinder REST API requests

Allow the core API to cache resources, such as DB results, so that
extensions can use data already retrieved within the same API request
eliminating additional expensive DB calls.

Loosely based on commit 9f9fbc54e7336da10fc3056bdaca2ec7d01c7f94 from
nova.

Change-Id: If9f49faf7305287c0489ad6209cf19b8bec612cc
Partial-Bug: #1197612
(cherry picked from commit 233430b51ddbd207f22eee98386ffbb766fd777f)
Use cached volumes in REST API extensions

This patch caches db volumes in the core volumes API so that they can be
used in the host and tenant attribute API extensions in order to lower
the response time by avoiding costly individual db queries for each
volume in the REST API contributions.

Change-Id: Ie0cbe929379df32fddbdce1953f13b6f26208bff
Partial-Bug: #1197612
(cherry picked from commit 9ed0f39fceaaa7a56e74055b54b7f6124e9e7f11)
GlusterFS: Synchronize operations that manipulate qcow2 data

Operations that modify qcow2 chains or the GlusterFS driver's
snapshot info file should use locking so that more than one thread
cannot run these operations at the same time.

Without this it is possible for multiple threads to concurrently
modify snapshot information resulting in an incomplete snapshot
chain.

Closes-Bug: #1251425

(cherry picked from commit 8a1fd8889170c289910944db016a4206fe5e1e32)
Change-Id: I93c6ffecd9aec38560cb19085243390b9120b363
Continue to delete volumes that DNE in rbd backend

If a volume has already been deleted in the RBD backend, catch the
exception and let Cinder continue removing it in the database.

Closes-Bug: #1254318
Change-Id: I99b591ce78d82c6eaabff0459d9c523e85c8e65f
(cherry picked from commit 3de7da12d1098ef777305099e5f4a039e536bf99)
Don't do glance v2 calls when config is set to v1

Fixes bug 1252549

Change-Id: I6d9509bfecd2f73040a0ad27aff8fc5f6fd0a899
(cherry picked from commit a5424f700e6cce409763019906acfe293a75ed1a)
GlusterFS: Set correct permissions for volume file created via clone

This is currently done when creating a new volume from scratch,
but not when cloning a new volume.

(cherry picked from commit 9c816aa0cf20ff0200831f58d8f14efb5e9e288e)
Closes-Bug: #1254768
Change-Id: I8506dc0824bacdb6cc42b4cadd41c2cad3a74b5b
Pass the size when fetching image in xenapi driver

In commit 6e287c0f2bb7d4994d50f1763f412277e4dac6f7 the fetch_to_raw
method was changed to accept the volume size so it can fail when the
image doesn't fit in the destination volume. In this commit we also
updated all the volume drivers to pass the volume size to that method so
it could perform the check. The xenapi driver missed that change.

This commit updates the xenapi driver to pass the volume size when
fetching the image to enable the introduced check.

Change-Id: I89bc4b3b1c1e6b06f5c529166cf9ffa1fe600c96
Related-Bug: #1235358
(cherry picked from commit c4db75abddbb72ac5f717f6a9266e078b69206a8)