Use pipe between ceph backup diff export/import

We now use a piped transfer between the rbd export-diff
and import-diff for incremental backups/restores as
opposed to holding the entire diff in memory.

Change-Id: I33476d9b3934781413af5cd2867a11d825a5d78e
Fixes: bug 1244464
(cherry picked from commit d384d28e1c2c7c3ce0bd676fb1c2bdb8a2d98a03)
Fail when image is bigger than the volume

When creating a new volume from a qcow2 image stored in glance, we will
be given the physical size of the image instead of the virtual size.
Most drivers will convert that image to raw after downloading for
resizing it to the requested volume size afterwards.

If the virtual size of the image is bigger than the requested one, the
resulting volume might end up being unusable after the resize even
though the creation went good.

This patch will make the volume creation fail if the virtual size of
the image exceeds the one of the requested volume.

Closes-Bug: #1235358
Change-Id: I254cd9e3acf5d9eb7eb913b37d27a14e97568dec
(cherry picked from commit 6e287c0f2bb7d4994d50f1763f412277e4dac6f7)
Add call to retrieve image metadata for volumes in bulk

When using the GET volume details REST API call, the image metadata API
contribution is making an individual db call for each of the available
volumes. When the number of volumes is large the details call can take
several minutes.

This patch adds a call to the volume.API to retrieve image metadata in
bulk, very similar to the one used to retrieve individual volume image
metadata.

Change-Id: Ic3aa721016704c72b7564cc5ceff71676806a24a
Partial-Bug: #1197612
(cherry picked from commit bb2daca2864a5607635a3aa37f3874fdda22797b)
Retrieve volume image metadata using single query

The image metadata REST API contributions are making an individual db
call for each of the available volumes. When the number of volumes is
large the volume details call can take several minutes.

This patch changes the image metadata API contributions to take
advantage of the new db query to retrieve metadata in bulk.

Change-Id: I9a35438c1f38ea8a3d8f5b687ae58ba1f3f78121
Partial-Bug: #1197612
(cherry picked from commit 56e5805203a14acad78dc806247b5f0f4a22c9f8)
Enable object caching in cinder REST API requests

Allow the core API to cache resources, such as DB results, so that
extensions can use data already retrieved within the same API request
eliminating additional expensive DB calls.

Loosely based on commit 9f9fbc54e7336da10fc3056bdaca2ec7d01c7f94 from
nova.

Change-Id: If9f49faf7305287c0489ad6209cf19b8bec612cc
Partial-Bug: #1197612
(cherry picked from commit 233430b51ddbd207f22eee98386ffbb766fd777f)
Use cached volumes in REST API extensions

This patch caches db volumes in the core volumes API so that they can be
used in the host and tenant attribute API extensions in order to lower
the response time by avoiding costly individual db queries for each
volume in the REST API contributions.

Change-Id: Ie0cbe929379df32fddbdce1953f13b6f26208bff
Partial-Bug: #1197612
(cherry picked from commit 9ed0f39fceaaa7a56e74055b54b7f6124e9e7f11)
GlusterFS: Synchronize operations that manipulate qcow2 data

Operations that modify qcow2 chains or the GlusterFS driver's
snapshot info file should use locking so that more than one thread
cannot run these operations at the same time.

Without this it is possible for multiple threads to concurrently
modify snapshot information resulting in an incomplete snapshot
chain.

Closes-Bug: #1251425

(cherry picked from commit 8a1fd8889170c289910944db016a4206fe5e1e32)
Change-Id: I93c6ffecd9aec38560cb19085243390b9120b363
Continue to delete volumes that DNE in rbd backend

If a volume has already been deleted in the RBD backend, catch the
exception and let Cinder continue removing it in the database.

Closes-Bug: #1254318
Change-Id: I99b591ce78d82c6eaabff0459d9c523e85c8e65f
(cherry picked from commit 3de7da12d1098ef777305099e5f4a039e536bf99)
Don't do glance v2 calls when config is set to v1

Fixes bug 1252549

Change-Id: I6d9509bfecd2f73040a0ad27aff8fc5f6fd0a899
(cherry picked from commit a5424f700e6cce409763019906acfe293a75ed1a)
GlusterFS: Set correct permissions for volume file created via clone

This is currently done when creating a new volume from scratch,
but not when cloning a new volume.

(cherry picked from commit 9c816aa0cf20ff0200831f58d8f14efb5e9e288e)
Closes-Bug: #1254768
Change-Id: I8506dc0824bacdb6cc42b4cadd41c2cad3a74b5b
Pass the size when fetching image in xenapi driver

In commit 6e287c0f2bb7d4994d50f1763f412277e4dac6f7 the fetch_to_raw
method was changed to accept the volume size so it can fail when the
image doesn't fit in the destination volume. In this commit we also
updated all the volume drivers to pass the volume size to that method so
it could perform the check. The xenapi driver missed that change.

This commit updates the xenapi driver to pass the volume size when
fetching the image to enable the introduced check.

Change-Id: I89bc4b3b1c1e6b06f5c529166cf9ffa1fe600c96
Related-Bug: #1235358
(cherry picked from commit c4db75abddbb72ac5f717f6a9266e078b69206a8)
Merge "Fail when image is bigger than the volume" into stable/havana
Merge "Pass the size when fetching image in xenapi driver" into stable/havana
Merge "Continue to delete volumes that DNE in rbd backend" into stable/havana
Fix Storwize terminate_connection with no host

Nova may pass a connector to Cinder with no 'host' field, which was
causing a KeyError in the Storwize driver. This patch resolves this case
by doing the following:
1. If the volume is mapped to only 1 host, unmap it
2. If the volume was not mapped or mapped to multiple hosts, print a
   warning but don't raise an exception

Change-Id: I0ec1c24adbdfbcf1c4868b4981a2e2618d4b411c
Closes-Bug: #1244257
(cherry picked from commit 8185d1b5db421441ebba128f23f840a42f9bf050)
LVM: Activate source LV before cloning from it

LVM may be configured to not automatically activate
thin-provisioned LVs.

Ensure they are activated when performing a clone, otherwise
dd will fail as the device does not exist in /dev/mapper/.

Closes-Bug: #1252423

Change-Id: Ibcb946ffe7804b1976bf1b1863c48340c8cc7fd5
(cherry picked from commit b9ff8cd7ad8c402787324d2baca9b32f61eafb4a)
Merge "GlusterFS: Synchronize operations that manipulate qcow2 data" into stable/havana
Merge "GlusterFS: Set correct permissions for volume file created via clone" into stable/havana
Merge "Fix Storwize terminate_connection with no host" into stable/havana
Merge "Use pipe between ceph backup diff export/import" into stable/havana