vmware:Ignore inaccessible/inMaintenance datastore

The vmdk driver uses the number of hosts connected to a datastore as one
of the criteria for selecting a datastore for volume creation. It might
incorrectly consider a host to which the datastore is inaccessible while
computing the number of connected hosts. This is because, the driver
sometimes uses the aggregated accessible status of a datastore while
determining its accessibility to a host. Also, the driver is ignoring
the maintenance status of the datastore and might create a volume in a
datastore which is entering maintenance or already in maintenance. This
change fixes these issues in datastore filtering.

Change-Id: Ibd5c1865d42638a735fd588e9c10e8f714c9c030
Closes-Bug: #1291346
Closes-Bug: #1291291
Add XML deserializer for qos_manage delete_keys API

XML deserializer is missing for delete_keys API resulting XML format
request not being correctly handled by QoS Manage API extension. This
patch adds XML deserializer for the API.

Also add unittest for qos_manage XML serializer and deserilizer.

Change-Id: I7b3ac6822c52f11b08d767aa55b7107bd0333c36
Closes-bug: #1312553
Sync periodic_task from oslo-incubator

This is more or less to get commit c63fd5a from oslo into the core
projects which have several periodic tasks.  Cinder has periodic tasks
to sync up volume state with the server and they don't have specific
spacing values set which can lead to non-deterministic spacing of when
the tasks run.

Note that this does not include the gettextutils and log dependencies
since there are not functional changes in those modules needed for the
periodic_task changes synced in *and* more importantly, the changes
to gettextutils and log require pervasive changes to cinder which
should happen when cinder integrates with the oslo-i18n library for
blueprint i18n--messages.

Further note that this does not include jsonutils due to some
issues introduced with a change for python 2.6 that impacts how strings
are encoded with simplejson. The details for that issue are in bug
1314129.  The jsonutils changes are not related to the periodic_task
changes being synced in so the dependency is not functionally required.

cinder.db.base.Base is also updated otherwise multiple inheritance
involving that class will not work, which impacts all of the classes
that extend cinder.manager.Manager which extends both cinder.db.Base
and PeriodicTasks, and commit 47c9d60 adds attributes to the __init__
for PeriodicTasks.  Nova had the same change with commit 5ae97ea.

Changes:

c63fd5a Make unspecified periodic spaced tasks run on default interval
f0dd798 Remove rendundant parentheses of cfg help strings
fcf517d Update oslo log messages with translation domains
051b9f3 Refactor unnecessary arithmetic ops in periodic_task
674cdaf Refactor if logic in periodic_task
b6b82c5 Use timestamp in periodic tasks
47c9d60 Don't share periodic_task instance data in a class attr
8b2b0b7 Use hacking import_exceptions for gettextutils._
c5a1088 Typos fix in db and periodic_task module
12bcdb7 Remove vim header

Related-Bug: #1319232

Change-Id: I13e8ac83bcd9e60a8eb05ed9cdfea00b9e5fb398
Remove check_{attach,detach} from volumes API

These methods are not called from anywhere and are likely just a relic
from splitting Cinder out from Nova. Moreover their usage from the API
should really be discouraged as they are inherently racy (the result is
immediately outdated since it is not atomic with anything that might
follow).

Partial-bug: #1302774
Change-Id: I5d79b22b2a1d3c9be429a13d094328fc2075fcd2
Fix log capturing fixture

Instead of nuking the unit test logs capture them according
to how tempest (and oslotest) capture them so that the tests
when ran locally provide the full logging output instead of
a miniaturized version.

This adds in the following environment variables which can
be used to control testr captured logging:

- OS_LOG_CAPTURE (turns on log capturing so that the testr
  repository will have log output, at a INFO level)
- OS_DEBUG (changes the captured logging level from INFO
  to DEBUG, only useful when OS_LOG_CAPTURE is enabled)

It also creates a lambda that is used to test whether an
environment is enabled (avoiding recreating the same set
of checks in multiple places).

Change-Id: I63d9a1b7fc6e7591fd6309cf68d9269117c023a6
Add strip size support to rbd driver

Rbd support strip size for volume, in general, we would like to think the
default value 4M is a proper for most of users.

In the ceph side, the proper value of rbd strip size can improve performance for
the special OSD backend.

Implements blueprint add-rbd-strip-size

Change-Id: I44d89681a78bd70d8c185e3da7d58f59e6076899
Signed-off-by: Haomai Wang <haomai@unitedstack.com>
Remove unused oslo module and adjust opentstack-common.conf

The issues are detected by new change about update.py
in I68c0108304be8a315bcefe939c22019caab9bc98. This patch
doesn't depend the oslo-incubator patch.
 * remove unused module crypto
 * add missing module loopingcall in openstack-common.conf
 * sort the modules in openstack-common.conf

Change-Id: I9f0fcd3803f89a6902f92d0ea691b9ae0235f64c
vmware: Fixes VMDK volume incompatibility issue

VMDK volume created in a cluster having ESXi 5.1 host has a high
hardware version i.e. 9; hence, fails when attached to instance
on an ESXi 5.0 host. This patch sets the hardware version to 8,
so that the volume can be attached to instances on any ESXi host
starting version 5.0 and above.

Change-Id: Ia750fd1183334005a834eb49be767451d87b338c
Closes-Bug: #1325586
Retry lvremove with ignore_suspended_devices

A lvremove -f might leave behind suspended devices
when it is racing with udev or other processes
still accessing any of the device files. The previous
solution of using lvchange -an on the LV had the
side-effect of deactivating origin LVs alongway in
the thick volume case, which was undesired.

It turns out retrying the deactivation twice and
ignoring the suspended devices on the second iteration
avoids the hang of all LVM operations after an initial
failure.

Change-Id: I0d6fb74084d049ea184e68f2dcc4e74f400b7dbd
Closes-Bug: #1317075
Related-Bug: #1270192
Add task/flow listener support around the volume api flow

This will add in logging of what is occurring around the
actions taken by taskflow when taskflow executes the initial
create_volume workflow.

Since the oslo logging module will include the request_id of
the currently active request (per greenthread) this will make
it easier to understand and debug the actions occurring inside
taskflow's engine concept.

Part of blueprint task-logging

Change-Id: I57a6f85ecac37fc4f97033eb3b7a0dc8bd35a886
Add keyword argument missing at some exc.HTTPError subclass

Some exc.HTTPError subclass needs keyword argument ``explanation``.
If there is no ``explanation=``, the message of the argument is
recognized as ``detail``.

Change-Id: Idfc07b2bd71da31e44212d3296921e18d2024a63
Closes-Bug: #1264424
glusterfs: Honor mount options when restarting cinder service

When restarting cinder service (cinder-volume specifically),
ensure that the gluster mounts are unmounted and remounted so
that any new mount options added to the shares config file
is taken effect, post service restart.

Change-Id: I9fe3442cd4899770b4c9ea4a73a7fe62967f2c64
I/O rate limit for volume copy with dd

Currently, volume copy operations consumes disk I/O bandwidth heavily and
may slow down the other guest instances. This patch limits bandwidth for
volume copy to mitigate interference to other instance performance.

In this implementation, 'dd' is put in a blkio cgroup for throttling,
when CONF.volume_copy_bps_limit is set to non-zero.

Change-Id: I32347d2e28842725207a1b3afac7fe137a3fd3b4
Signed-off-by: Tomoki Sekiyama <tomoki.sekiyama@hds.com>
Implements: blueprint limit-volume-copy-bandwidth
I/O rate limit for volume copy with qemu-img convert

Currently, volume copy operations consumes disk I/O bandwidth heavily and
may slow down the other guest instances. This patch limits bandwidth for
volume copy to mitigate interference to other instance performance.

In this implementation, 'qemu-img' is put in a blkio cgroup for throttling,
when CONF.volume_copy_bps_limit is set to non-zero.

Change-Id: I8403d4bcafc4279654034da97aa9b6c097f9e649
Signed-off-by: Tomoki Sekiyama <tomoki.sekiyama@hds.com>
Implements: blueprint limit-volume-copy-bandwidth
Handle the case where az is disabled/removed

Instead of retaining a local in-memory list of
availability zones that is never purged and therefore
never updates itself if availability zones are removed
we should instead cache those zones for a given period
of time (defaulting to one hour) and on cache expiry
refetch the enabled availability zones.

DocImpact: adds a new configuration that defines the
default availability zone cache expiry time in seconds.

This will enable the engine that runs the create volume
request to in a future change remove the need to have
a functor which proxies the validation logic around
availability zones.

Change-Id: I8e45845b66ba62313248ee9cbeb301a5fe54a08b
Fixes an issue with 'dd' bug from Illumos repo

The Illumos version of 'dd' never stops in case of count=0.
Thus 'truncate' should be used for creating a sparsed file.

Change-Id: I7a1840e2c43a27c648e5f65c2a87286311a34516
Merge "Add strip size support to rbd driver"
Implement extend volume in NFS driver

This fix implements the extend_volume functionality in the NFS
driver.

Change-Id: I1634fef0206b4ef12684c6c7f4851d76da579942
Closes-Bug: #1295316
Merge "Remove unused oslo module and adjust opentstack-common.conf"
debug level logs should not be translated

According to the OpenStack translation policy available at
https://wiki.openstack.org/wiki/LoggingStandards debug messages
should not be translated. Like mentioned in several changes in
Nova by garyk this is to help prioritize log translation.

This patch adds a new hacking check - N319 - that ensures all
debug log messages don't have translations.

Change-Id: Id9c2715f25c8f2ea52235aba4bd1583655391584
Implements: blueprint debug-translation-removal
Closes-Bug: #1318713