Improve Cinder API internal cache interface

Improve the internal caching system used by the API layer
by borrowing the implementation used by Nova.

Unlike the previous implementation, this new interface makes it clear
which resource types are added or retrieved from the cache.

This change also adds other resources to the cache:
- backups
- snapshots
- volume types

It's now possible to remove database access those extensions:
- extended_snapshot_attributes
- volume_host_attribute
- volume_mig_status_attribute
- volume_tenant_attribute

Closes-bug: #1358524
Change-Id: I1fc673d0c01c41faa98292d5813d4471b455d712
Fix unnecessary snap of glance image, with non-raw images

When the glance image format is not raw, the glance image stored on
GPFS should not be snapped. Because, we copy the image to cinder volume
repository converting the image format to 'raw'. So, no copy_on_write
is supported.

Change-Id: I9cc3ffb2f36b65876b05ecb9f4654461be4516b7
Closes-Bug: #1378964
Add client_socket_timeout option

Add a parameter to take advantage of the new(ish) eventlet socket timeout
behaviour.  Allows closing idle client connections after a period of
time, eg:

$ time nc localhost 8776
real    1m0.063s

Setting 'client_socket_timeout = 0' means do not timeout.

DocImpact

Change-Id: If492810a2f10fa5954f8c8bb708b14be0b77fb90
Closes-bug: #1371022
Log a warning when getting lvs and vgs takes longer than 60 seconds

We know something is causing lvs/vgs commands to block while deleting a
volume and this is causing Tempest to timeout while waiting for the
volume to be deleted.  What we don't have right now is very good
(specific) logging when this happens, unless we get messages in syslog
for lvm tasks taking more than 120 seconds, but that doesn't always
happen when we see the volume delete timeout in Tempest.

This patch adds a check for when getting logical volumes and volume
groups takes longer than 60 seconds and logs a warning if that happens.
This is helpful in production also because the default interval for
periodic tasks is 60 seconds so having these take longer than that time
could cause periodic tasks to block up on each other and you'll get
warnings from the FixedIntervalLoopingCall in oslo which is controlling
the task runs.

Related-Bug: #1373513

Change-Id: I7ac5ddbbd9bda7744db33f0bb8d56879301f5538
Merge "Fix unnecessary snap of glance image, with non-raw images"
Merge "Log a warning when getting lvs and vgs takes longer than 60 seconds"
Merge "Add client_socket_timeout option"
Turn on Flake-8 Complexity Checking

Flake8 provides the ability to measure code complexity. There are
a lot of modules in Cinder that are considered "too complex", the
worst being "cinder/tests/test_huawei_hvs.py:110:1:" with a complexity
ranking of 59.

There's some outlyers at the higher end here, but the majority of the
code checks in at under 30, so let's make that our threshold and ignore
the two offenders that are above that for now.

Granted this may or may not be valuable, but it doesn't hurt to try it
and if we all hate it or find there's no value but it makes life difficult
we can always turn it back off.

See flake8.readthedocs for more info on flake8 and McCabe complexity
checking.

Change-Id: Icc1b5f2ccf0b630c1e655f5e6abf4e25ee30afc4
Merge "Turn on Flake-8 Complexity Checking"
Merge "Improve Cinder API internal cache interface"
Move volume copy logic into manager

TODO: explanation is necessary

Related-Blueprint: generic-volume-migration
Change-Id: Ic3add11c272d3e262cc9a1a640660d947ea6f855
Introduce changes necessary for generic volume migration

This patch contains the core driver changes needed to make generic
volume migration possible.  By itself, no changes to the existing logic
for volume migration are introduced.  The intent is to keep the original
behaviour intact while allowing certain drivers that cannot attach
volumes locally to still participate in volume migration.

To allow this functionality, an 'is_attachable' function is added to the
volume driver definition.  This property indicates whether the volume
can be attached locally to the host and expose a path to the resulting
block device.  A driver author can override this and return False if
a driver cannot attach its volumes in a way that exposes a block device
path.  Doing so will cause the volume copy routine (copy_volume_data) to
use file I/O instead of calling out to `dd`.  The driver author is
expected to provide an implementation of open_volume() that returns
a file object capable of proxying I/O requests to the underlying storage
backend.

An RPC call for 'is_attachable' is added to allow querying of remote
volume capabilities.

The default volume open and close routines are currently unimplemented
and are the next step in this series.  The open routine will return
a file handle defined by the path to the block device.  This will allow
drivers like RBD which do not support attachment to migrate to and from
drivers which do support attachment, such as LVM.

Related-Blueprint: generic-volume-migration
Change-Id: I5876f755bc1ea972749fdcc02e1f186f7f2b207d
WIP

Change-Id: Ifcfccd96b869d22dd667c8e09ad2bc6586b30a32
WIP

Change-Id: I84a8022ce76e13c53022969cc77d5c6427e0566e
WIP

Change-Id: If2bbbedf73f28a48e8fcc8e6868fb531acf38589
WIP

Change-Id: I995d4a8dbc7ed887a6fff260b79e3880536f638f
WIP

Change-Id: I26681675456b722e50d54091cc4bb3eb41218114