vmware:Ignore inaccessible/inMaintenance datastore

The vmdk driver uses the number of hosts connected to a datastore as one
of the criteria for selecting a datastore for volume creation. It might
incorrectly consider a host to which the datastore is inaccessible while
computing the number of connected hosts. This is because, the driver
sometimes uses the aggregated accessible status of a datastore while
determining its accessibility to a host. Also, the driver is ignoring
the maintenance status of the datastore and might create a volume in a
datastore which is entering maintenance or already in maintenance. This
change fixes these issues in datastore filtering.

Change-Id: Ibd5c1865d42638a735fd588e9c10e8f714c9c030
Closes-Bug: #1291346
Closes-Bug: #1291291
CinderException args to strings when exceptions

CinderException message and kwargs are converted to strings if the
argument is itself an exception.

Exceptions passed as arguments to CinderException happen in many areas
of the code. This causes an error when oslo.messaging tries to serialise
the exception.

Change-Id: I1399c939bbca47ab8362ac3bbe0e9a349c6d5572
Closes-Bug: #1301249
Use oslo network utils function to set tcp_keepalive

Earlier, tcp_keepidle was set to 600s and other parameters left to default
values. A new function was added to oslo-incubator for setting tcp keepalive
settings, and cinder should use this to set any values we want.
In addition to the existing cfg option for tcp_keepidle, new values were added
for tcp_keepalive(to turn on/off this feature, tcp_keepintvl(to specify gap
between successive probes) and tcp_keepcnt(to specify number of probes). To
mimic existing behavior, tcp_keepalive defaults to True, and tcp_keepidle to
600(seconds). The other two default to None, and the oslo code will ignore
them.
Ran update.py in oslo-incubator to copy changed modules into cinder.
Changes are:
network_utils.py (which contains the function we need, others are deps)
edd73c9 Merge "Improve help strings"
fe3389e Improve help strings
8a0f567 Remove str() from LOG.* and exceptions
f61a4e7 Fix python26 compatibility for RFCSysLogHandler
da6d713 Revert setting oslo-incubator logs to INFO
0d18381 Set default log levels for oslo.messaging and oslo-incubator

gettextutils.py
9912e5d Add API for creating translation functions
6cc96d0 Fix test_gettextutils on Python 3

jsonutils.py
9e5a393 jsonutils.load() accepts file pointer, not string
a6b2aec On Python <= 2.6, use simplejson if available
e3a1d9c Use six.moves.xmlrpc_client instead of xmlrpclib
3a31bba Python3 support for xmlrpclib
e6494c2 Use six.iteritems to make dict work on Python2/3

log.py
edd73c9 Merge "Improve help strings"
fe3389e Improve help strings
8a0f567 Remove str() from LOG.* and exceptions
f61a4e7 Fix python26 compatibility for RFCSysLogHandler
da6d713 Revert setting oslo-incubator logs to INFO
0d18381 Set default log levels for oslo.messaging and oslo-incubator

Current HEAD in OSLO:
----------------------------------------------
Merge: 2640847 e433899
Date:   Mon May 5 19:41:28 2014 +0000
Merge "Enable configuring tcp keepalive"
----------------------------------------------

Ran generate_sample to generate cinder.conf.sample

Implements blueprint configure-tcp-keepalive
Change-Id: I2e4c2327803c25e54ed771f25ef973bb543b9355
Limit formatting routes when adding resources

By default, routes.mapper.Mapper.resource adds a bunch of formatted
routes that accept anything after a '.'. Our spec says only .xml
and .json are accepted so limit the formatting to those valuse.

This allows identifiers with a '.' in urls. A few tests were added
to the extensions test to prove that .xml and .json are stripped
but other values are not.

Closes-bug: #1307956
Change-Id: I8f0b4ae3b10ea93890f6506662905bd5823412f5
Add XML deserializer for qos_manage delete_keys API

XML deserializer is missing for delete_keys API resulting XML format
request not being correctly handled by QoS Manage API extension. This
patch adds XML deserializer for the API.

Also add unittest for qos_manage XML serializer and deserilizer.

Change-Id: I7b3ac6822c52f11b08d767aa55b7107bd0333c36
Closes-bug: #1312553
Remove second get call to list/show volumes

Recently it was observed that with large numbers of volumes
that things like "cinder list" could take extremely long to
return.

For example, running cinder list on a system with 1000 volumes took
greater than 30 seconds to return.  It turns out that the cause of
this is the addition of visible admin_metadata.

There's two problems with this:
1. The original patch probably shouldn't have gone in
   data is either admin data or it's not, selectively picking
   pieces of admin data out to provide to the user just creates
   complications and introduces confusion.
2. The REAL issue here is that since the standar gets are made
   with the standard user context, the add_visible_admin_metadata
   would go through and do an elevated context get on every single
   volume individually.  This is what caused the horrible performance
   issue on cinder list with large numbers of volumes.

Running as admin, or removing the second call drops this down to about 3
seconds for the same 1000 volume list.

This patch removes the secondary admin context get_call.  Instead where
we expect to do display the visible admin_meta, we pass in a flag
requesting that the volume object we're getting has the appropriate
metadata.  This way we can elevate the context if needed and avoid
iterating through the gets again.

This patch also cleans up the get_visible_admin_meta methods, and
consolidates both V1 and V2 to use the utils method.

Change-Id: I3fb7aefb7d8a5664b0a3fb3958f509b5cd621320
Closes-Bug: 1317606
Sync periodic_task from oslo-incubator

This is more or less to get commit c63fd5a from oslo into the core
projects which have several periodic tasks.  Cinder has periodic tasks
to sync up volume state with the server and they don't have specific
spacing values set which can lead to non-deterministic spacing of when
the tasks run.

Note that this does not include the gettextutils and log dependencies
since there are not functional changes in those modules needed for the
periodic_task changes synced in *and* more importantly, the changes
to gettextutils and log require pervasive changes to cinder which
should happen when cinder integrates with the oslo-i18n library for
blueprint i18n--messages.

Further note that this does not include jsonutils due to some
issues introduced with a change for python 2.6 that impacts how strings
are encoded with simplejson. The details for that issue are in bug
1314129.  The jsonutils changes are not related to the periodic_task
changes being synced in so the dependency is not functionally required.

cinder.db.base.Base is also updated otherwise multiple inheritance
involving that class will not work, which impacts all of the classes
that extend cinder.manager.Manager which extends both cinder.db.Base
and PeriodicTasks, and commit 47c9d60 adds attributes to the __init__
for PeriodicTasks.  Nova had the same change with commit 5ae97ea.

Changes:

c63fd5a Make unspecified periodic spaced tasks run on default interval
f0dd798 Remove rendundant parentheses of cfg help strings
fcf517d Update oslo log messages with translation domains
051b9f3 Refactor unnecessary arithmetic ops in periodic_task
674cdaf Refactor if logic in periodic_task
b6b82c5 Use timestamp in periodic tasks
47c9d60 Don't share periodic_task instance data in a class attr
8b2b0b7 Use hacking import_exceptions for gettextutils._
c5a1088 Typos fix in db and periodic_task module
12bcdb7 Remove vim header

Related-Bug: #1319232

Change-Id: I13e8ac83bcd9e60a8eb05ed9cdfea00b9e5fb398
hp_lefthand_rest_proxy no handler for logger during tests

Create the proxy during do_setup() instead of __init__()
so that the logger is ready for an import error during tests.

Closes-Bug: #1325116
Change-Id: I88145d97df5d746e92bc8302bf516ca9eaa05d00
Remove check_{attach,detach} from volumes API

These methods are not called from anywhere and are likely just a relic
from splitting Cinder out from Nova. Moreover their usage from the API
should really be discouraged as they are inherently racy (the result is
immediately outdated since it is not atomic with anything that might
follow).

Partial-bug: #1302774
Change-Id: I5d79b22b2a1d3c9be429a13d094328fc2075fcd2
LIO: Don't add ACL for local initiator name

Currently cinder-rtstool requires /etc/iscsi/initiatorname.iscsi
to be present in order to create a LUN/ACL/portal successfully.
However, this is not required since Cinder will create the
required ACLs dynamically at attach time as well.

Removing this path means that iscsi-initiator-utils no longer
needs to be installed locally when using a remote Nova compute
node.

Closes-Bug: 1328580

Change-Id: Ieb5e28e3f81987ab8518b861fff885f371969a56
Fix Brocade FC driver's use of the fc_fabric_names

The Brocade Fibre Channel Zone Manager driver wasn't
stripping out whitespace in the fc_fabric_names
prior to using them.  This would cause breakage later on
in the driver.

Change-Id: I6d16a35026fdd2f30a92f9dff16435a6c39b699a
Closes-Bug: 1328681
Fix log capturing fixture

Instead of nuking the unit test logs capture them according
to how tempest (and oslotest) capture them so that the tests
when ran locally provide the full logging output instead of
a miniaturized version.

This adds in the following environment variables which can
be used to control testr captured logging:

- OS_LOG_CAPTURE (turns on log capturing so that the testr
  repository will have log output, at a INFO level)
- OS_DEBUG (changes the captured logging level from INFO
  to DEBUG, only useful when OS_LOG_CAPTURE is enabled)

It also creates a lambda that is used to test whether an
environment is enabled (avoiding recreating the same set
of checks in multiple places).

Change-Id: I63d9a1b7fc6e7591fd6309cf68d9269117c023a6
Add strip size support to rbd driver

Rbd support strip size for volume, in general, we would like to think the
default value 4M is a proper for most of users.

In the ceph side, the proper value of rbd strip size can improve performance for
the special OSD backend.

Implements blueprint add-rbd-strip-size

Change-Id: I44d89681a78bd70d8c185e3da7d58f59e6076899
Signed-off-by: Haomai Wang <haomai@unitedstack.com>
Copy custom properties to image from volume

Presently after copying an image to volume, all properties
of the image are getting copied properly but while creating
image back from volume, it doesn't copy custom properties
to the image.

At present in volume-glance-metadata table all the properties
of volume are stored as key and value. Because of this it is
difficult to differentiate between core and custom properties.
To overcome this, I have added a new option
'glance_core_properties' in cinder.conf. This option defines
all core properties of an image. This way, it's easy to separate
core and custom properties from the glance_volume_metadata and add
custom property to the newly created image.

For Example:
glance_core_properties = 'checksum', 'container_format',
'disk_format', 'image_name', 'image_id', 'min_disk',
'min_ram', 'name', 'size'

DocImpact: Added 'glance_core_properties' to distinguish the
core and custom properties as discussed above in cinder.conf

blueprint: restrict-uploading-volume-to-image

Change-Id: I786edbc6e54b3d06ef679a71e22676d2f88e7307
Remove unused oslo module and adjust opentstack-common.conf

The issues are detected by new change about update.py
in I68c0108304be8a315bcefe939c22019caab9bc98. This patch
doesn't depend the oslo-incubator patch.
 * remove unused module crypto
 * add missing module loopingcall in openstack-common.conf
 * sort the modules in openstack-common.conf

Change-Id: I9f0fcd3803f89a6902f92d0ea691b9ae0235f64c
vmware: Fixes VMDK volume incompatibility issue

VMDK volume created in a cluster having ESXi 5.1 host has a high
hardware version i.e. 9; hence, fails when attached to instance
on an ESXi 5.0 host. This patch sets the hardware version to 8,
so that the volume can be attached to instances on any ESXi host
starting version 5.0 and above.

Change-Id: Ia750fd1183334005a834eb49be767451d87b338c
Closes-Bug: #1325586
Merge "hp_lefthand_rest_proxy no handler for logger during tests"
Implements HDS-Cinder HNAS Drivers

This change introduces HDS HNAS iSCSI and NFS Drivers.

HNAS NFS certification results:
  https://gist.github.com/sombrafam/6c73cb823ce75b0538bd

HNAS iSCSI certification results:
  https://gist.github.com/sombrafam/2625987243ec91d53c7a

Change-Id: I1a16f76887cf22dcf8ce0ed1dbc0b57327a87616
Implements: blueprint hds-hnas
Merge "Copy custom properties to image from volume"
Allow reset-state on attach and migration fields

The reset-state API call only sets that main status
column on the volume object.  There's also a method
to set attach status, however it turns out that due
to the implementation of the status validation method
in the parent class, this was impossible to use because
it only checked and accepted "status" NOT "attach_status".

This patch fixes that attach_status problem, it also implements
the ability to update the migration status.  A cinderclient change
will be needed to expose these calls as well.

DocImpact

Change-Id: I59e1bb2522f033c944fa07acf4876ca71c8c3d3f
Closes-Bug: #1308253