Improve Cinder API internal cache interface

Improve the internal caching system used by the API layer
by borrowing the implementation used by Nova.

Unlike the previous implementation, this new interface makes it clear
which resource types are added or retrieved from the cache.

This change also adds other resources to the cache:
- backups
- snapshots
- volume types

It's now possible to remove database access those extensions:
- extended_snapshot_attributes
- volume_host_attribute
- volume_mig_status_attribute
- volume_tenant_attribute

Closes-bug: #1358524
Change-Id: I1fc673d0c01c41faa98292d5813d4471b455d712
Avoid using the disk cache on volume initialisation

When caching is involved, the volume might be
unmapped before the copy actually hit the disk or the
VM starts booting before all of the data has been flushed,
which causes the VM to crash at an arbitrary point in time.

Closes-Bug: #1363016

Change-Id: I7a04f683add8c23b9125fe837c4048ccc3ac224d
Fix unnecessary snap of glance image, with non-raw images

When the glance image format is not raw, the glance image stored on
GPFS should not be snapped. Because, we copy the image to cinder volume
repository converting the image format to 'raw'. So, no copy_on_write
is supported.

Change-Id: I9cc3ffb2f36b65876b05ecb9f4654461be4516b7
Closes-Bug: #1378964
Fix running unit tests with coverage

The path to this repo is added to PYTHONPATH when the coverage flag is
specified so that the cinder entry points are discovered.

Change-Id: Ie33a63c6ecd443b900fa7bed3a1ec5b6bb0065e6
Closes-Bug: #1364347
Some tcp configuration paramters are ignored

The values for tcp_keepalive, tcp_keepidle, tcp_keepalive_count and
tcp_keepalive_interval in cinder.conf are not being picked up.

Change-Id: Ie0a232ae6b9b9a34d57b48f3b0d8694f90504517
Closes-bug: #1366011
Use right body for test_create_missing_specs_name

Since the test is `test_create_missing_specs_name`, we should define
the the body like `body = {'qos_specs': {'a': 'b'}}`. And add a test
to cover the invalid body case.

Change-Id: Ib1cf9a8b5bfddc0091e28e3765a36c75044642f3
Closes-bug: #1370308
Revert iSCSI Target objects as independent objects

Removing from gate queue since failing unit tests

The second half of this code didn't make it in, so this is essentially
dead code. We'll readd this code back in Kilo.

This reverts commit d10fb796f491152486e394481e920479bab9e3d1.

Closes-Bug: #1365516
Change-Id: Ia4b3b4ef4a7b6f3829fe576ec370b6d74a7bc922
Fix confusing exception message in NetApp iscsi driver

Fix confusing exception message when no target details
are returned on volume attach.

Change-Id: Ib5fe15a4a6be14a20b1ae446b8b57737b50e9aeb
Closes-Bug: 1366816
HP 3PAR configurable ssh-host-key-policy

Cinder added configurable SSH strict key policy (RejectPolicy)
and configurable known_hosts_file.  This patch configures the
hp3parclient (used by the 3PAR drivers) using those settings.

Change-Id: I71fa7d8be28d1cb7c0102590e03cba994b6035a0
Closes-bug: 1365697
SQL scripts should not manage transactions

Cinder SQL scripts assume too much about the environment and the way in
which they are executed by sqlalchemy-migrate. Particularly, they manage
transactions on their own even though sqlalchemy-migrate do it itself,
plus it assumes that scripts are executed in one go (even though
standard Python DB-API 2.0 does not mention multi-statement SQL commands
to be executed on DB cursors.

blueprint enable-mysql-connector

Change-Id: I10c58b3af75d3ab9153a8bbd2a539bf1577de328
Closes-Bug: 1368391
Stop using intersphinx

Remove intersphinx from the docs build as it triggers network calls that
occasionally fail, and we don't really use intersphinx (links other
sphinx documents out on the internet)

This also removes the requirement for internet access during docs build.

This causes docs jobs to fail because we error out on warnings.

Change-Id: I71e941e2a639641a662a163c682eb86d51de42fb
Related-Bug: #1368910
Volume types need to be specified when creating CG

When creating a consistency group, the scheduler will find a backend that
supports all input volume types. If volume types are not provided, the
default_volume_type in cinder.conf will be used, however, this could cause
inconsistent behavior in a user environment where the default_volume_type
is defined in some places but not in others. This fix removed the use of
default_volume_type for CG creation, added a check to verify that volume
types are provided when creating a CG.

When creating a volume and adding it to a CG, we need to make sure a
volume type is provided as well.

Change-Id: I078d4fdd8d92529e853be16272ad74d1e130f712
Closes-Bug: #1366371
Handle config file with newlines and whitespaces

EMC VMAX driver does not handle newlines and whitespaces
consistently in the XML config file. This patch addressed
this issue.

Change-Id: I45f61f167feee9174ba2bc3229695b680f8e946d
Closes-Bug: #1364232
Getting iscsi_ip_address from cinder.conf

In the current version (2.0) of the VMAX driver, we retrieve iSCSI IP
addresses dynamically from SMI-S. However, we ran into situations where
we can't get them reliably during testing. The fix is to get this
information from cinder.conf, just like in version 1.0.

Change-Id: If474f44e99cc45592bacfdec9e1eb2fa48c41e68
Closes-Bug: #1369355
Updated from global requirements

Change-Id: Ie85ed9ad591edc3e38352fe0c7ee751a43557a38
IBMNAS: Remove call to set r/w permissions to all

During cinder volume create operation from a volume snapshot or
from an existing volume (volume clone operation), the ibmnas
driver sets 'rw' permissions to all, which is unnecessary and
also poses security concerns.
Fixing this issue, removing the calls to set rw permissions to
all during these operations and adding a call to set 'rw'
permissions only to the owner to make sure even if umask is set
at the filesystem level, which might deny 'rw' access to the owner
we explicitely set the required permissions on the volume file.

Change-Id: I0e5ba9262a298e088f7724ddeda3537afa4b023e
Closes-Bug: #1367238
Fix a problem with 'volume list' when 'all_tenants=0'

This change checks the value of 'all_tenants' and returns a list of volumes
for all tenants when 'all_tenants' is 1 (or True). A tenant-specific volume
list is returned when 'all_tenants' is 0 (or False).

An InvalidInput exception is thrown if the 'all_tenants' value is not 0, 1,
True, or False (case insensitive).

Provided new unit tests for the get_all api which check the all_tenants, and
limits parameters.

Fixed pep8 compliance test fails with code being improperly indented, and
changed calls using str() to use six.text_type()

Change-Id: If0b8c343ee7e4e05c9aec89241458dbdf2e4734b
Closes-Bug: #1342046
Merge "HP 3PAR configurable ssh-host-key-policy"
Merge "Avoid using the disk cache on volume initialisation"
Remove executable bits on various files

During the Juno cycle we landed a number of commits
making files executable that shouldn't be.

Change-Id: If2c5279b263b35526a41f4f026386b75329ff9a7