Avoid using the disk cache on volume initialisation

When caching is involved, the volume might be
unmapped before the copy actually hit the disk or the
VM starts booting before all of the data has been flushed,
which causes the VM to crash at an arbitrary point in time.

Closes-Bug: #1363016

Change-Id: I7a04f683add8c23b9125fe837c4048ccc3ac224d
Fix running unit tests with coverage

The path to this repo is added to PYTHONPATH when the coverage flag is
specified so that the cinder entry points are discovered.

Change-Id: Ie33a63c6ecd443b900fa7bed3a1ec5b6bb0065e6
Closes-Bug: #1364347
Some tcp configuration paramters are ignored

The values for tcp_keepalive, tcp_keepidle, tcp_keepalive_count and
tcp_keepalive_interval in cinder.conf are not being picked up.

Change-Id: Ie0a232ae6b9b9a34d57b48f3b0d8694f90504517
Closes-bug: #1366011
Use right body for test_create_missing_specs_name

Since the test is `test_create_missing_specs_name`, we should define
the the body like `body = {'qos_specs': {'a': 'b'}}`. And add a test
to cover the invalid body case.

Change-Id: Ib1cf9a8b5bfddc0091e28e3765a36c75044642f3
Closes-bug: #1370308
Revert iSCSI Target objects as independent objects

Removing from gate queue since failing unit tests

The second half of this code didn't make it in, so this is essentially
dead code. We'll readd this code back in Kilo.

This reverts commit d10fb796f491152486e394481e920479bab9e3d1.

Closes-Bug: #1365516
Change-Id: Ia4b3b4ef4a7b6f3829fe576ec370b6d74a7bc922
Fix confusing exception message in NetApp iscsi driver

Fix confusing exception message when no target details
are returned on volume attach.

Change-Id: Ib5fe15a4a6be14a20b1ae446b8b57737b50e9aeb
Closes-Bug: 1366816
Stop using intersphinx

Remove intersphinx from the docs build as it triggers network calls that
occasionally fail, and we don't really use intersphinx (links other
sphinx documents out on the internet)

This also removes the requirement for internet access during docs build.

This causes docs jobs to fail because we error out on warnings.

Change-Id: I71e941e2a639641a662a163c682eb86d51de42fb
Related-Bug: #1368910
Volume types need to be specified when creating CG

When creating a consistency group, the scheduler will find a backend that
supports all input volume types. If volume types are not provided, the
default_volume_type in cinder.conf will be used, however, this could cause
inconsistent behavior in a user environment where the default_volume_type
is defined in some places but not in others. This fix removed the use of
default_volume_type for CG creation, added a check to verify that volume
types are provided when creating a CG.

When creating a volume and adding it to a CG, we need to make sure a
volume type is provided as well.

Change-Id: I078d4fdd8d92529e853be16272ad74d1e130f712
Closes-Bug: #1366371
Handle config file with newlines and whitespaces

EMC VMAX driver does not handle newlines and whitespaces
consistently in the XML config file. This patch addressed
this issue.

Change-Id: I45f61f167feee9174ba2bc3229695b680f8e946d
Closes-Bug: #1364232
Getting iscsi_ip_address from cinder.conf

In the current version (2.0) of the VMAX driver, we retrieve iSCSI IP
addresses dynamically from SMI-S. However, we ran into situations where
we can't get them reliably during testing. The fix is to get this
information from cinder.conf, just like in version 1.0.

Change-Id: If474f44e99cc45592bacfdec9e1eb2fa48c41e68
Closes-Bug: #1369355
Updated from global requirements

Change-Id: Ie85ed9ad591edc3e38352fe0c7ee751a43557a38
Fix a problem with 'volume list' when 'all_tenants=0'

This change checks the value of 'all_tenants' and returns a list of volumes
for all tenants when 'all_tenants' is 1 (or True). A tenant-specific volume
list is returned when 'all_tenants' is 0 (or False).

An InvalidInput exception is thrown if the 'all_tenants' value is not 0, 1,
True, or False (case insensitive).

Provided new unit tests for the get_all api which check the all_tenants, and
limits parameters.

Fixed pep8 compliance test fails with code being improperly indented, and
changed calls using str() to use six.text_type()

Change-Id: If0b8c343ee7e4e05c9aec89241458dbdf2e4734b
Closes-Bug: #1342046
Merge "Avoid using the disk cache on volume initialisation"
Remove executable bits on various files

During the Juno cycle we landed a number of commits
making files executable that shouldn't be.

Change-Id: If2c5279b263b35526a41f4f026386b75329ff9a7
Sync latest strutils from oslo-incubator for mask_password fix

This sync pulls in:
   1131b56 Enable mask_password to handle byte code strings

This is the only update since the last sync:
   beca4db Sync latest strutils from oslo-incubator

This is needed to fix a delete volume failure when processutils
calls mask_password with a byte code string.

Change-Id: Ie8a8c8e26abc0e387830bc9246e9a44913ae9b24
Closes-bug: #1368527
VMware: Improve datastore selection logic

The current datastore selection logic is not modularized and difficult
to extend. The retype API implementation has a requirement to specify
hard anti-affinity requirement with the current backing datastore. Some
of the bug fixes also need to specify hard affinity with one or more
datastore types. To support such requirements and to enable future
extensions, this patch introduces a new module which contains datastore
selection logic. The existing code for datastore selection is reused as
much as possible. The dependency on existing datastore selection logic
will be removed in a separate patch.

The current datastore selection iterates over a list of hosts, for each
host, queries the connected valid datastores and tries to select a
suitable datastore. The filtering is based on space, storage profile,
number of connected hosts and space utilization. The space utilization
is used only for breaking ties. If a suitable datastore is found, further
processing of list of hosts is skipped, which could result in uneven space
utilization. To solve this, the new selection logic introduces a requirement
called 'preferred_utilization_threshold' which can be exposed as a driver
config option.

Partial-bug: #1275682
Partial-bug: #1301943
Partial-bug: #1293955

Change-Id: I17e90aa09a303fbb8d4ad90037f440c8c4e7d072
VMware: Implement retype for VMDK driver

This patch adds support for retype operation in the VMDK driver.

The VMDK driver uses the extra spec keys 'vmware:vmdk_type' and
'vmware:storage_profile' to determine the disk provisioning type
and storage policy of the backing VM. Currently it is not possible
to change the disk provisioning type or storage policy after volume
creation. With retype support, user can change the disk provisioning
type and storage policy by changing the volume's type to another type
with extra spec keys set to desired values.

Closes-Bug: #1275682
Partial-Bug: #1288254

Change-Id: If4b868af7b91373b2b36fd5cd2f0dda12d604d99
Check replication status failed for non-replication

Need to check self.replication whether is None before the
function get_replication_status be called.

Closes-bug: #1365392

Change-Id: I6f6939a98091e40537f2355bb716b4b3205845e2
VMware: Add storage profile related unit tests

This patch adds missing unit tests related to storage profiles.

Change-Id: I8819437617d813fbabdb79dcdcaa529e8a30796a
NetApp fix for controller preferred path

This patch optimizes the data path by choosing
the preferred path to access the volume by selecting
the controller owning the volume.

Closes-Bug: #1365881

Change-Id: I7069111dfd22d4fca92615b5730854f6d696ec13