vmware:Ignore inaccessible/inMaintenance datastore

The vmdk driver uses the number of hosts connected to a datastore as one
of the criteria for selecting a datastore for volume creation. It might
incorrectly consider a host to which the datastore is inaccessible while
computing the number of connected hosts. This is because, the driver
sometimes uses the aggregated accessible status of a datastore while
determining its accessibility to a host. Also, the driver is ignoring
the maintenance status of the datastore and might create a volume in a
datastore which is entering maintenance or already in maintenance. This
change fixes these issues in datastore filtering.

Change-Id: Ibd5c1865d42638a735fd588e9c10e8f714c9c030
Closes-Bug: #1291346
Closes-Bug: #1291291
Use oslo network utils function to set tcp_keepalive

Earlier, tcp_keepidle was set to 600s and other parameters left to default
values. A new function was added to oslo-incubator for setting tcp keepalive
settings, and cinder should use this to set any values we want.
In addition to the existing cfg option for tcp_keepidle, new values were added
for tcp_keepalive(to turn on/off this feature, tcp_keepintvl(to specify gap
between successive probes) and tcp_keepcnt(to specify number of probes). To
mimic existing behavior, tcp_keepalive defaults to True, and tcp_keepidle to
600(seconds). The other two default to None, and the oslo code will ignore
them.
Ran update.py in oslo-incubator to copy changed modules into cinder.
Changes are:
network_utils.py (which contains the function we need, others are deps)
edd73c9 Merge "Improve help strings"
fe3389e Improve help strings
8a0f567 Remove str() from LOG.* and exceptions
f61a4e7 Fix python26 compatibility for RFCSysLogHandler
da6d713 Revert setting oslo-incubator logs to INFO
0d18381 Set default log levels for oslo.messaging and oslo-incubator

gettextutils.py
9912e5d Add API for creating translation functions
6cc96d0 Fix test_gettextutils on Python 3

jsonutils.py
9e5a393 jsonutils.load() accepts file pointer, not string
a6b2aec On Python <= 2.6, use simplejson if available
e3a1d9c Use six.moves.xmlrpc_client instead of xmlrpclib
3a31bba Python3 support for xmlrpclib
e6494c2 Use six.iteritems to make dict work on Python2/3

log.py
edd73c9 Merge "Improve help strings"
fe3389e Improve help strings
8a0f567 Remove str() from LOG.* and exceptions
f61a4e7 Fix python26 compatibility for RFCSysLogHandler
da6d713 Revert setting oslo-incubator logs to INFO
0d18381 Set default log levels for oslo.messaging and oslo-incubator

Current HEAD in OSLO:
----------------------------------------------
Merge: 2640847 e433899
Date:   Mon May 5 19:41:28 2014 +0000
Merge "Enable configuring tcp keepalive"
----------------------------------------------

Ran generate_sample to generate cinder.conf.sample

Implements blueprint configure-tcp-keepalive
Change-Id: I2e4c2327803c25e54ed771f25ef973bb543b9355
Add XML deserializer for qos_manage delete_keys API

XML deserializer is missing for delete_keys API resulting XML format
request not being correctly handled by QoS Manage API extension. This
patch adds XML deserializer for the API.

Also add unittest for qos_manage XML serializer and deserilizer.

Change-Id: I7b3ac6822c52f11b08d767aa55b7107bd0333c36
Closes-bug: #1312553
Sync periodic_task from oslo-incubator

This is more or less to get commit c63fd5a from oslo into the core
projects which have several periodic tasks.  Cinder has periodic tasks
to sync up volume state with the server and they don't have specific
spacing values set which can lead to non-deterministic spacing of when
the tasks run.

Note that this does not include the gettextutils and log dependencies
since there are not functional changes in those modules needed for the
periodic_task changes synced in *and* more importantly, the changes
to gettextutils and log require pervasive changes to cinder which
should happen when cinder integrates with the oslo-i18n library for
blueprint i18n--messages.

Further note that this does not include jsonutils due to some
issues introduced with a change for python 2.6 that impacts how strings
are encoded with simplejson. The details for that issue are in bug
1314129.  The jsonutils changes are not related to the periodic_task
changes being synced in so the dependency is not functionally required.

cinder.db.base.Base is also updated otherwise multiple inheritance
involving that class will not work, which impacts all of the classes
that extend cinder.manager.Manager which extends both cinder.db.Base
and PeriodicTasks, and commit 47c9d60 adds attributes to the __init__
for PeriodicTasks.  Nova had the same change with commit 5ae97ea.

Changes:

c63fd5a Make unspecified periodic spaced tasks run on default interval
f0dd798 Remove rendundant parentheses of cfg help strings
fcf517d Update oslo log messages with translation domains
051b9f3 Refactor unnecessary arithmetic ops in periodic_task
674cdaf Refactor if logic in periodic_task
b6b82c5 Use timestamp in periodic tasks
47c9d60 Don't share periodic_task instance data in a class attr
8b2b0b7 Use hacking import_exceptions for gettextutils._
c5a1088 Typos fix in db and periodic_task module
12bcdb7 Remove vim header

Related-Bug: #1319232

Change-Id: I13e8ac83bcd9e60a8eb05ed9cdfea00b9e5fb398
Remove check_{attach,detach} from volumes API

These methods are not called from anywhere and are likely just a relic
from splitting Cinder out from Nova. Moreover their usage from the API
should really be discouraged as they are inherently racy (the result is
immediately outdated since it is not atomic with anything that might
follow).

Partial-bug: #1302774
Change-Id: I5d79b22b2a1d3c9be429a13d094328fc2075fcd2
LIO: Don't add ACL for local initiator name

Currently cinder-rtstool requires /etc/iscsi/initiatorname.iscsi
to be present in order to create a LUN/ACL/portal successfully.
However, this is not required since Cinder will create the
required ACLs dynamically at attach time as well.

Removing this path means that iscsi-initiator-utils no longer
needs to be installed locally when using a remote Nova compute
node.

Closes-Bug: 1328580

Change-Id: Ieb5e28e3f81987ab8518b861fff885f371969a56
Fix log capturing fixture

Instead of nuking the unit test logs capture them according
to how tempest (and oslotest) capture them so that the tests
when ran locally provide the full logging output instead of
a miniaturized version.

This adds in the following environment variables which can
be used to control testr captured logging:

- OS_LOG_CAPTURE (turns on log capturing so that the testr
  repository will have log output, at a INFO level)
- OS_DEBUG (changes the captured logging level from INFO
  to DEBUG, only useful when OS_LOG_CAPTURE is enabled)

It also creates a lambda that is used to test whether an
environment is enabled (avoiding recreating the same set
of checks in multiple places).

Change-Id: I63d9a1b7fc6e7591fd6309cf68d9269117c023a6
Add strip size support to rbd driver

Rbd support strip size for volume, in general, we would like to think the
default value 4M is a proper for most of users.

In the ceph side, the proper value of rbd strip size can improve performance for
the special OSD backend.

Implements blueprint add-rbd-strip-size

Change-Id: I44d89681a78bd70d8c185e3da7d58f59e6076899
Signed-off-by: Haomai Wang <haomai@unitedstack.com>
Remove unused oslo module and adjust opentstack-common.conf

The issues are detected by new change about update.py
in I68c0108304be8a315bcefe939c22019caab9bc98. This patch
doesn't depend the oslo-incubator patch.
 * remove unused module crypto
 * add missing module loopingcall in openstack-common.conf
 * sort the modules in openstack-common.conf

Change-Id: I9f0fcd3803f89a6902f92d0ea691b9ae0235f64c
vmware: Fixes VMDK volume incompatibility issue

VMDK volume created in a cluster having ESXi 5.1 host has a high
hardware version i.e. 9; hence, fails when attached to instance
on an ESXi 5.0 host. This patch sets the hardware version to 8,
so that the volume can be attached to instances on any ESXi host
starting version 5.0 and above.

Change-Id: Ia750fd1183334005a834eb49be767451d87b338c
Closes-Bug: #1325586
Retry lvremove with ignore_suspended_devices

A lvremove -f might leave behind suspended devices
when it is racing with udev or other processes
still accessing any of the device files. The previous
solution of using lvchange -an on the LV had the
side-effect of deactivating origin LVs alongway in
the thick volume case, which was undesired.

It turns out retrying the deactivation twice and
ignoring the suspended devices on the second iteration
avoids the hang of all LVM operations after an initial
failure.

Change-Id: I0d6fb74084d049ea184e68f2dcc4e74f400b7dbd
Closes-Bug: #1317075
Related-Bug: #1270192
Add task/flow listener support around the volume api flow

This will add in logging of what is occurring around the
actions taken by taskflow when taskflow executes the initial
create_volume workflow.

Since the oslo logging module will include the request_id of
the currently active request (per greenthread) this will make
it easier to understand and debug the actions occurring inside
taskflow's engine concept.

Part of blueprint task-logging

Change-Id: I57a6f85ecac37fc4f97033eb3b7a0dc8bd35a886
Removes unecessary call to rbd.Image

Removed duplicate and unprotected call to rbd.Image
in rbd driver which can result in an uncaught exception
if the rbd image no longer exists.

Also added unit test for _delete_backup_snaps()

Closes-Bug: 1323203
Change-Id: I89eedb0d0276a985b4d496093df2eee2ce067ec1
Made provision for providing optional arguments

The 'quota_committed' attribute of 'RequestContext' object is
a transient property, so it will not be saved in the taskflow
persistent storage. The updated value of 'quota_committed'
attribute will not be available while resuming/reverting the
flow, if cinder api-service is down/stopped after committing
the quota.

Since this 'quota_committed' attribute is not used anywhere
in cinder project other than in create-volume taskflow api, so
removed 'quota_committed' from RequestContext and made
provision to pass it as an optional argument which will be
passed to api-flow via create_what dictionary, in order to
make it persistent and use it as and when needed.

Change-Id: I719067a8e1824add45fb525c106642a9c0e6bb46
Merge "Use oslo network utils function to set tcp_keepalive"
Merge "Made provision for providing optional arguments"
glusterfs: Honor mount options when restarting cinder service

When restarting cinder service (cinder-volume specifically),
ensure that the gluster mounts are unmounted and remounted so
that any new mount options added to the shares config file
is taken effect, post service restart.

Change-Id: I9fe3442cd4899770b4c9ea4a73a7fe62967f2c64
Merge "LIO: Don't add ACL for local initiator name"
Handle the case where az is disabled/removed

Instead of retaining a local in-memory list of
availability zones that is never purged and therefore
never updates itself if availability zones are removed
we should instead cache those zones for a given period
of time (defaulting to one hour) and on cache expiry
refetch the enabled availability zones.

DocImpact: adds a new configuration that defines the
default availability zone cache expiry time in seconds.

This will enable the engine that runs the create volume
request to in a future change remove the need to have
a functor which proxies the validation logic around
availability zones.

Change-Id: I8e45845b66ba62313248ee9cbeb301a5fe54a08b
Fixes an issue with 'dd' bug from Illumos repo

The Illumos version of 'dd' never stops in case of count=0.
Thus 'truncate' should be used for creating a sparsed file.

Change-Id: I7a1840e2c43a27c648e5f65c2a87286311a34516