Use pipe between ceph backup diff export/import

We now use a piped transfer between the rbd export-diff
and import-diff for incremental backups/restores as
opposed to holding the entire diff in memory.

Change-Id: I33476d9b3934781413af5cd2867a11d825a5d78e
Fixes: bug 1244464
(cherry picked from commit d384d28e1c2c7c3ce0bd676fb1c2bdb8a2d98a03)
NetApp fix free space as zero during 1st vol stats update

NetApp clustered ontap drivers report space as zero
till first 60 seconds of driver start. This is causing
discomfort for some performance sensitive deployements.
This was due to async nature of the NetApp stats collection job.
Job is changed to sync at driver start to improve customer
experience.

Change-Id: I7d5cbf590897a0d328ece3a60516c92c0ad0ee7f
Closes-bug: #1253660
(cherry picked from commit 585f34ff7798ad271121d432e0cae8820bc54389)
Add call to retrieve image metadata for volumes in bulk

When using the GET volume details REST API call, the image metadata API
contribution is making an individual db call for each of the available
volumes. When the number of volumes is large the details call can take
several minutes.

This patch adds a call to the volume.API to retrieve image metadata in
bulk, very similar to the one used to retrieve individual volume image
metadata.

Change-Id: Ic3aa721016704c72b7564cc5ceff71676806a24a
Partial-Bug: #1197612
(cherry picked from commit bb2daca2864a5607635a3aa37f3874fdda22797b)
Retrieve volume image metadata using single query

The image metadata REST API contributions are making an individual db
call for each of the available volumes. When the number of volumes is
large the volume details call can take several minutes.

This patch changes the image metadata API contributions to take
advantage of the new db query to retrieve metadata in bulk.

Change-Id: I9a35438c1f38ea8a3d8f5b687ae58ba1f3f78121
Partial-Bug: #1197612
(cherry picked from commit 56e5805203a14acad78dc806247b5f0f4a22c9f8)
Enable object caching in cinder REST API requests

Allow the core API to cache resources, such as DB results, so that
extensions can use data already retrieved within the same API request
eliminating additional expensive DB calls.

Loosely based on commit 9f9fbc54e7336da10fc3056bdaca2ec7d01c7f94 from
nova.

Change-Id: If9f49faf7305287c0489ad6209cf19b8bec612cc
Partial-Bug: #1197612
(cherry picked from commit 233430b51ddbd207f22eee98386ffbb766fd777f)
Use cached volumes in REST API extensions

This patch caches db volumes in the core volumes API so that they can be
used in the host and tenant attribute API extensions in order to lower
the response time by avoiding costly individual db queries for each
volume in the REST API contributions.

Change-Id: Ie0cbe929379df32fddbdce1953f13b6f26208bff
Partial-Bug: #1197612
(cherry picked from commit 9ed0f39fceaaa7a56e74055b54b7f6124e9e7f11)
Brick connector revised fix for NFS drivers

This change fixes the error that comes
while getting brick connector and attaching volumes
in case of NFS drivers in cinder. The attribute for
mount point base was not passed to attach_volume method
as the connector initialization logic is common for all
types of protocols. It is fixed by populating
the required parameter in the RemoteFsConnector
by extracting it from connection properties. The infrastructure
for fixing this situation for remote fs drivers is implemented
in this fix. The drivers need to override the method
_get_mount_point_base in their implementation. NFS driver
implements it. GlusterFs implementation of the method to follow
in a separate patch. Hence marking it partial fix.

Change-Id: Id61a437fca1870529fa8b85f7fc2f73eae665294
Partial-Bug: #1238085
(cherry picked from commit 1cd362630655798e5ff4c058b3f44862ae61c801)
GlusterFS: Synchronize operations that manipulate qcow2 data

Operations that modify qcow2 chains or the GlusterFS driver's
snapshot info file should use locking so that more than one thread
cannot run these operations at the same time.

Without this it is possible for multiple threads to concurrently
modify snapshot information resulting in an incomplete snapshot
chain.

Closes-Bug: #1251425

(cherry picked from commit 8a1fd8889170c289910944db016a4206fe5e1e32)
Change-Id: I93c6ffecd9aec38560cb19085243390b9120b363
Don't do glance v2 calls when config is set to v1

Fixes bug 1252549

Change-Id: I6d9509bfecd2f73040a0ad27aff8fc5f6fd0a899
(cherry picked from commit a5424f700e6cce409763019906acfe293a75ed1a)
GlusterFS: Set correct permissions for volume file created via clone

This is currently done when creating a new volume from scratch,
but not when cloning a new volume.

(cherry picked from commit 9c816aa0cf20ff0200831f58d8f14efb5e9e288e)
Closes-Bug: #1254768
Change-Id: I8506dc0824bacdb6cc42b4cadd41c2cad3a74b5b
Fix Storwize terminate_connection with no host

Nova may pass a connector to Cinder with no 'host' field, which was
causing a KeyError in the Storwize driver. This patch resolves this case
by doing the following:
1. If the volume is mapped to only 1 host, unmap it
2. If the volume was not mapped or mapped to multiple hosts, print a
   warning but don't raise an exception

Change-Id: I0ec1c24adbdfbcf1c4868b4981a2e2618d4b411c
Closes-Bug: #1244257
(cherry picked from commit 8185d1b5db421441ebba128f23f840a42f9bf050)
LVM: Activate source LV before cloning from it

LVM may be configured to not automatically activate
thin-provisioned LVs.

Ensure they are activated when performing a clone, otherwise
dd will fail as the device does not exist in /dev/mapper/.

Closes-Bug: #1252423

Change-Id: Ibcb946ffe7804b1976bf1b1863c48340c8cc7fd5
(cherry picked from commit b9ff8cd7ad8c402787324d2baca9b32f61eafb4a)
GlusterFS: Ensure Cinder can write to shares

Ensure the Cinder user can write to the GlusterFS share.  This
is required for snapshot functionality, and means the admin
does not have to set this permission manually.

Conflicts:
	cinder/tests/test_glusterfs.py

Closes-Bug: #1236966
Change-Id: I4a9ea40df9681ca6931ad6b390aa21b09d6cfec9
(cherry picked from commit 371fa540600b20b97eae389e1f976145866cadae)
Merge "GlusterFS: Synchronize operations that manipulate qcow2 data" into stable/havana
Merge "GlusterFS: Set correct permissions for volume file created via clone" into stable/havana
Merge "Fix Storwize terminate_connection with no host" into stable/havana
Merge "Use pipe between ceph backup diff export/import" into stable/havana
Merge "Don't do glance v2 calls when config is set to v1" into stable/havana
Sync rpc fix from oslo-incubator

Sync the following fixes from oslo-incubator:

ef406a2 Create a shared queue for QPID topic consumers
e227c0e Properly reconnect subscribing clients when QPID broker restarts

Closes-bug: #1251757
Closes-bug: #1257293
Change-Id: I934a6ea6e9eb510513640870750ea37e6f076df9
Merge "Add call to retrieve image metadata for volumes in bulk" into stable/havana