Fixes 3par driver methods that were double locking

There were many cases where the 3par drivers(FC and iSCSI) were double
locking, e.g. we were locking in the individual driver and in ‘common’.
This patch set removes all the locks in ‘common’ except the one on
‘create_cloned_volume’.

The methods ’_get_volume_state’ and ‘_copy_volume’ are only called in
‘create_cloned_volume’, so those locks were removed as well. All the other
methods in ‘common’ are only called by the driver which provide the lock,
except ‘create_volume’ which is also called by ‘create_cloned_volume’.

Change-Id: I660fbd04824bd2837e06480c8298ea5a709fc82a
Fixes: bug 1175791
NetApp direct to filer drivers for iscsi and nfs.

The drivers are designed to work with 7 mode and cluster storage systems.
The drivers are designed to communicate with the NetApp storage controllers
directly without requiring any intermediate management softwares in place using
the NetApp api. Hence these represent the direct to filer versions of NetApp drivers.
This also contains conventional nfs driver for cluster systems which requires
NetApp management software as middle layer. Implementation for cloning support
and basic volume type scheduler support in driver added. All NetApp drivers
shifted into separate NetApp package. NetApp api refactored into a separate file
other than drivers.

blueprint netapp-direct-volume-drivers
blueprint netapp-cluster-nfs-driver

Change-Id: If3f5389d1f98eba899c4928a8ae1c1d3430bd847
Upgrade WebOb to 1.2.3

The version of WebOb being used in OpenStack was more than
1 year old. This change updates to the latest stable release.

Upgrading WebOb resolves a version conflict between OpenStack
and Pecan, the web framework used by the Ceilometer team for
version 2 of the ceilometer API.

Refer to http://docs.webob.org/en/latest/news.html
for the list of changes between 1.0.8 and 1.2.3.

bug 1092227

Change-Id: I44fdd11acbf1d5245e0c9c5a6199fd53200928ed
Signed-off-by: Doug Hellmann <doug.hellmann@dreamhost.com>
Use tempdir for lock_path in tests.

Commit: 2331d3336a6adf4fc13a3b187e91a5d1b1f7c723
introduced use of lockutils but wasn't setting a lock_path, in the tests.
As a result we were polluting cinder/openstack/ with cinder-xxxx lockfiles.

This patch adds a tempfile and cleanup to the test so we don't leave any
junk behind.

Change-Id: Idd9fee928ff0fcf3f8d1c30ee88c5306b88e290c
Implement filter scheduler

In order to do more sophisticated scheduling (e.g. schedule based on volume
type), filter scheduler is introduced. Several changes are made to make this
possible, some of them are similar to the counterpart in Nova:

- add HostState class to host_manager in order to store volume capabilities
- implement get_volume_stats() method of iSCSIDriver as an example to
demonstrate how volume backend driver reports capabilities as well as status
- add scheduler_options.py and 'scheduler_json_config_location' flag to be
allow loading json configuration file for scheduler at run time
- port common filters/weights from oslo
- add capacity weigher (scheduler/weights/capacity.py) for picking up
target volume backend by weighing free capacity of the host. The default
behavior is to spread volumes across hosts; by changing the
'capacity_weight_multiplier' to negative number, volume placing behavior will
become stacking.
- add capacity filter which filters those hosts have insufficient storage space
to serve the request.
- add 'reserved_percentage' config option to indicate how much space is
reserved. Capacity reservation is needed when volume resize is enabled.
- change default scheduler to 'filter.FilterScheduler'.
- add 'publish_service_capabilities()' method to volume RPC API to allow
scheduler to have volume service report capabilities. This bumps volume RPC
API to version 1.2
- remove 'volume_force_update_capabilities' config option, volume status will be
report to scheduler in every checking.

The implication of this change to storage/backend driver developer:
- implementation of get_volume_stats() of the driver is now a *MUST*, filter
scheduler heavily relies on the status/capabilities reported by backend driver
to makeplacement decision.  To ensure Cinder works seamlessly on the storage
system, driver should at least report following capabilities/status:
----------------------+---------------------------+---------------------------
  Capability/Status   |      Description          |         Example
----------------------+---------------------------+---------------------------
 'volume_backend_name'| back-end name, string     | 'Example_Storage_Backend'
----------------------+---------------------------+---------------------------
  'vendor_name'       | vendor name, string       | 'OpenStackCinder'
----------------------+---------------------------+---------------------------
  'driver_version'    | version, string           |  '1.0a'
----------------------+---------------------------+---------------------------
  'storage_protocol'  | supported protocols,      | 'iSCSI', 'RBD', 'FC', 'NFS'
                      | string or list of strings | ['iSCSI', 'NFS', 'FC']
----------------------+---------------------------+---------------------------
  'total_capacity_gb' | capacity in GB, integer   |  102400
----------------------+---------------------------+---------------------------
  'free_capacity_gb'  | available capacity in GB, |  1000
                      | integer                   |
----------------------+---------------------------+---------------------------
'reserved_percentage' | reserved space in         |  0, 10
                      | percentage, integer       |
----------------------+---------------------------+---------------------------

The implication of this change to Cinder administrator:
- the default setting for filter scheduler should work well with the benefits
of:
  * being able to fully utilize capacity of backends (driver now has to report
  actul total space and space utilization and scheduler uses these info) not
  limited by the 'max_gigabytes' config option any more;
  * being able to choose placement policy between spreading & stacking for
  volume creation (by modifying the 'capacity_weight_multiplier' in
  CapacityWeigher)
- with filter scheduler, Cinder is now able to utilize the advanced features/
capabilities provided by different storage back-ends via: defining different
volume types with proper extra_specs. Volume types can be considered as sets
of back-end capabilities requirement.
 For example, a volume type which has 'storage_protocol':'FC' key/value pair
definition in its extra_spec can only be served by those back-ends who report
they support FiberChannel protocol. Another example is volume type has 'QoS'
requirement can only be served by back-ends support QoS.

Note/TODO:
* Currently scheduler makes its decision based on the status and capabilities
information reported by volume nodes, and these information is stored in memory
of scheduler process. More sophisticated way may be add on table in DB to
record status/capabilities of all volume nodes, like Nova does for compute nodes.

implement bp volume-type-scheduler

DocImpact

Change-Id: I8b5305bce3b24bcc85365037356a8f36d0107133
Add option to make exception format errors fatal.

Adds a new fatal_exception_format_errors config option which
defaults to False. This option is use to control how the base
CinderException class handles errors which can occur when it
formats error messages.

The motivation for this change is to be able to enable exception
format checking in our tests by setting fatal_exception_format_errors=True.

Change-Id: I2df952558d3f30a557cbb094f8bddcc37af9b5aa
Update exceptions to pass correct kwargs.

Updates a variety of Cinder exceptions so that they pass the correct
kwargs for proper exception message formatting. Previously
these exceptions would work but have incorrect error messages
because of how CinderException handled exception formatting errors.

Change-Id: Id1d15b112cad46a698227fa172d81a9ab586e66e
Enable cinder exception format checking in tests.

Updates the Cinder test runner so that it enables
'fatal_exception_format_errors' for testing.

Includes a bunch of fixes to test exceptions as well so they
will continue to pass.

Change-Id: Idd30a810fb81e8e14490644779c3e03b6af25ff3
Fix CinderClient exception name in EMCISCSIDriver.

Change NovaException to CinderException in the emc driver.

Change-Id: Id84ca2a354afc6a7b795aef779a9fdd2567ce0d6
Provide HP 3PAR array iSCSI driver

implements blueprint hp3par-volume-driver

We have the driver broken into 2 files:
hp_3par_common.py and
hp_3par_iscsi.py

The reason we do this is because we have a fibre channel driver
that will be submitted shortly after this is committed.   The
fibre channel driver and the iscsi driver share a lot of the same
code that talks to the 3PAR array for provisioning.  So,
it made sense not to have duplicate code.  The fibre channel driver
will be dependent on the fibre channel support I am actively working
on for nova/cinder grizzly release.

The driver uses a 2 mechanisms to talk to the 3PAR array:
1) a python REST client (hp3parclient) that lives in the pypi
   repository here:
   http://pypi.python.org/pypi/hp3parclient

2) SSH.  We had to pull in some of the ssh code from the base san
   driver to help fix an issue with executing commands on the 3PAR
   array.  The 3PAR has the ability to turn on CSV output for command
   results, which makes this easier to parse.  Unfortunately, there
   is no way to turn CSV mode on permanently for all ssh requests.
   So, we have to turn on the CSV output for every single ssh command
   issued.  Since we use ssh as well, we require the san_* options
   to be set.

We use a dual mechianism because the REST API that ships with the 3.1.2
firmware doesn't support all of the capabilities a cinder driver needs
to export volumes.

When a newer version of the firmware comes out that supports host
management on the 3PAR array, then we will get rid of the SSH code.

Change-Id: I9826ba1a36e27a9be05457ee9236a491dbfd0713
Merge "Add option to make exception format errors fatal."
Merge "Update exceptions to pass correct kwargs."
Merge "Enable cinder exception format checking in tests."
Merge "Use tempdir for lock_path in tests."
Merge "Provide HP 3PAR array iSCSI driver"
Merge "Implement filter scheduler"
Update SolidFire Volume driver

Adds support for clone volumes and set QoS use volume type / extra_specs.
Also includes some minor cleanup and some hardening for known pre-release
XDB issue in SolidFire Cluster API.

Change-Id: I4b39b301b9a1f9b859a694decfee48c74d22a74a
Revert "Implement filter scheduler"

This reverts commit 8dd2140c4c03582965d74232c7b61aef34e43661
Merge "Revert "Implement filter scheduler""
Implement filter scheduler

In order to do more sophisticated scheduling (e.g. schedule based on volume
type), filter scheduler is introduced. Several changes are made to make this
possible, some of them are similar to the counterpart in Nova:

- add HostState class to host_manager in order to store volume capabilities
- implement get_volume_stats() method of iSCSIDriver as an example to
demonstrate how volume backend driver reports capabilities as well as status
- add scheduler_options.py and 'scheduler_json_config_location' flag to be
allow loading json configuration file for scheduler at run time
- port common filters/weights from oslo
- add capacity weigher (scheduler/weights/capacity.py) for picking up
target volume backend by weighing free capacity of the host. The default
behavior is to spread volumes across hosts; by changing the
'capacity_weight_multiplier' to negative number, volume placing behavior will
become stacking.
- add capacity filter which filters those hosts have insufficient storage space
to serve the request.
- add 'reserved_percentage' config option to indicate how much space is
reserved. Capacity reservation is needed when volume resize is enabled.
- add 'publish_service_capabilities()' method to volume RPC API to allow
scheduler to have volume service report capabilities. This bumps volume RPC
API to version 1.2
- remove 'volume_force_update_capabilities' config option, volume status will be
report to scheduler in every checking.

The implication of this change to storage/backend driver developer:
- implementation of get_volume_stats() of the driver is now a *MUST*, filter
scheduler heavily relies on the status/capabilities reported by backend driver
to makeplacement decision.  To ensure Cinder works seamlessly on the storage
system, driver should at least report following capabilities/status:
----------------------+---------------------------+---------------------------
  Capability/Status   |      Description          |         Example
----------------------+---------------------------+---------------------------
 'volume_backend_name'| back-end name, string     | 'Example_Storage_Backend'
----------------------+---------------------------+---------------------------
  'vendor_name'       | vendor name, string       | 'OpenStackCinder'
----------------------+---------------------------+---------------------------
  'driver_version'    | version, string           |  '1.0a'
----------------------+---------------------------+---------------------------
  'storage_protocol'  | supported protocols,      | 'iSCSI', 'RBD', 'FC', 'NFS'
                      | string or list of strings | ['iSCSI', 'NFS', 'FC']
----------------------+---------------------------+---------------------------
  'total_capacity_gb' | capacity in GB, integer   |  102400
----------------------+---------------------------+---------------------------
  'free_capacity_gb'  | available capacity in GB, |  1000
                      | integer                   |
----------------------+---------------------------+---------------------------
'reserved_percentage' | reserved space in         |  0, 10
                      | percentage, integer       |
----------------------+---------------------------+---------------------------

The implication of this change to Cinder administrator:
- the default setting for filter scheduler should work well with the benefits
of:
  * being able to fully utilize capacity of backends (driver now has to report
  actul total space and space utilization and scheduler uses these info) not
  limited by the 'max_gigabytes' config option any more;
  * being able to choose placement policy between spreading & stacking for
  volume creation (by modifying the 'capacity_weight_multiplier' in
  CapacityWeigher)
- with filter scheduler, Cinder is now able to utilize the advanced features/
capabilities provided by different storage back-ends via: defining different
volume types with proper extra_specs. Volume types can be considered as sets
of back-end capabilities requirement.
 For example, a volume type which has 'storage_protocol':'FC' key/value pair
definition in its extra_spec can only be served by those back-ends who report
they support FiberChannel protocol. Another example is volume type has 'QoS'
requirement can only be served by back-ends support QoS.

Note/TODO:
* Currently scheduler makes its decision based on the status and capabilities
information reported by volume nodes, and these information is stored in memory
of scheduler process. More sophisticated way may be add on table in DB to
record status/capabilities of all volume nodes, like Nova does for compute nodes.

implement bp volume-type-scheduler

DocImpact

Change-Id: I296b3727db8de0d4cf085fac602d122a7b474842