NetApp direct to filer drivers for iscsi and nfs.

The drivers are designed to work with 7 mode and cluster storage systems.
The drivers are designed to communicate with the NetApp storage controllers
directly without requiring any intermediate management softwares in place using
the NetApp api. Hence these represent the direct to filer versions of NetApp drivers.
This also contains conventional nfs driver for cluster systems which requires
NetApp management software as middle layer. Implementation for cloning support
and basic volume type scheduler support in driver added. All NetApp drivers
shifted into separate NetApp package. NetApp api refactored into a separate file
other than drivers.

blueprint netapp-direct-volume-drivers
blueprint netapp-cluster-nfs-driver

Change-Id: If3f5389d1f98eba899c4928a8ae1c1d3430bd847
Upgrade WebOb to 1.2.3

The version of WebOb being used in OpenStack was more than
1 year old. This change updates to the latest stable release.

Upgrading WebOb resolves a version conflict between OpenStack
and Pecan, the web framework used by the Ceilometer team for
version 2 of the ceilometer API.

Refer to http://docs.webob.org/en/latest/news.html
for the list of changes between 1.0.8 and 1.2.3.

bug 1092227

Change-Id: I44fdd11acbf1d5245e0c9c5a6199fd53200928ed
Signed-off-by: Doug Hellmann <doug.hellmann@dreamhost.com>
Update SolidFire Volume driver

Adds support for clone volumes and set QoS use volume type / extra_specs.
Also includes some minor cleanup and some hardening for known pre-release
XDB issue in SolidFire Cluster API.

Change-Id: I4b39b301b9a1f9b859a694decfee48c74d22a74a
Revert "Implement filter scheduler"

This reverts commit 8dd2140c4c03582965d74232c7b61aef34e43661
Merge "Revert "Implement filter scheduler""
Implement filter scheduler

In order to do more sophisticated scheduling (e.g. schedule based on volume
type), filter scheduler is introduced. Several changes are made to make this
possible, some of them are similar to the counterpart in Nova:

- add HostState class to host_manager in order to store volume capabilities
- implement get_volume_stats() method of iSCSIDriver as an example to
demonstrate how volume backend driver reports capabilities as well as status
- add scheduler_options.py and 'scheduler_json_config_location' flag to be
allow loading json configuration file for scheduler at run time
- port common filters/weights from oslo
- add capacity weigher (scheduler/weights/capacity.py) for picking up
target volume backend by weighing free capacity of the host. The default
behavior is to spread volumes across hosts; by changing the
'capacity_weight_multiplier' to negative number, volume placing behavior will
become stacking.
- add capacity filter which filters those hosts have insufficient storage space
to serve the request.
- add 'reserved_percentage' config option to indicate how much space is
reserved. Capacity reservation is needed when volume resize is enabled.
- add 'publish_service_capabilities()' method to volume RPC API to allow
scheduler to have volume service report capabilities. This bumps volume RPC
API to version 1.2
- remove 'volume_force_update_capabilities' config option, volume status will be
report to scheduler in every checking.

The implication of this change to storage/backend driver developer:
- implementation of get_volume_stats() of the driver is now a *MUST*, filter
scheduler heavily relies on the status/capabilities reported by backend driver
to makeplacement decision.  To ensure Cinder works seamlessly on the storage
system, driver should at least report following capabilities/status:
----------------------+---------------------------+---------------------------
  Capability/Status   |      Description          |         Example
----------------------+---------------------------+---------------------------
 'volume_backend_name'| back-end name, string     | 'Example_Storage_Backend'
----------------------+---------------------------+---------------------------
  'vendor_name'       | vendor name, string       | 'OpenStackCinder'
----------------------+---------------------------+---------------------------
  'driver_version'    | version, string           |  '1.0a'
----------------------+---------------------------+---------------------------
  'storage_protocol'  | supported protocols,      | 'iSCSI', 'RBD', 'FC', 'NFS'
                      | string or list of strings | ['iSCSI', 'NFS', 'FC']
----------------------+---------------------------+---------------------------
  'total_capacity_gb' | capacity in GB, integer   |  102400
----------------------+---------------------------+---------------------------
  'free_capacity_gb'  | available capacity in GB, |  1000
                      | integer                   |
----------------------+---------------------------+---------------------------
'reserved_percentage' | reserved space in         |  0, 10
                      | percentage, integer       |
----------------------+---------------------------+---------------------------

The implication of this change to Cinder administrator:
- the default setting for filter scheduler should work well with the benefits
of:
  * being able to fully utilize capacity of backends (driver now has to report
  actul total space and space utilization and scheduler uses these info) not
  limited by the 'max_gigabytes' config option any more;
  * being able to choose placement policy between spreading & stacking for
  volume creation (by modifying the 'capacity_weight_multiplier' in
  CapacityWeigher)
- with filter scheduler, Cinder is now able to utilize the advanced features/
capabilities provided by different storage back-ends via: defining different
volume types with proper extra_specs. Volume types can be considered as sets
of back-end capabilities requirement.
 For example, a volume type which has 'storage_protocol':'FC' key/value pair
definition in its extra_spec can only be served by those back-ends who report
they support FiberChannel protocol. Another example is volume type has 'QoS'
requirement can only be served by back-ends support QoS.

Note/TODO:
* Currently scheduler makes its decision based on the status and capabilities
information reported by volume nodes, and these information is stored in memory
of scheduler process. More sophisticated way may be add on table in DB to
record status/capabilities of all volume nodes, like Nova does for compute nodes.

implement bp volume-type-scheduler

DocImpact

Change-Id: I296b3727db8de0d4cf085fac602d122a7b474842
Factor out LVM code.

Currently volume drivers inherit from VolumeDriver and ISCSIDriver,
which contain LVM-specific code. The LVM-specific functions are
generally overridden. The problem is that there is no place for
generic code for all drivers to inherit, which the LVM driver can
override. This patch basically makes the VolumeDriver and ISCSIDriver
classes mostly empty and moves the LVM-specific code to lvm.py. Also,
moved the global _iscsi_location() and _iscsi_authentication()
functions into the new LVMISCSIDriver class since nobody else used
them.

Change-Id: I067c975de97913bdc39086ad203cddef0c393c7a
Implements: blueprint factor-out-lvm-code
Enhance wsgi to listen on ipv6 address

Check if the hostname is ipv6 and set the family appropriately.
Picking up the snippet from glance.
Picked up some code from nova as well to get the test case running
properly

Change-Id: Ic79fcda2371f0907d75a142ea18b26b3e8d92e51
Support for SSL in wsgi.Server

SSL are entirely optional. Support for SSL as well using code from glance. We
have some new options for configuring the SSL support. There are tests for
accessing a sample app w/o ipv6 or ssl, one with just ssl and one with
ipv6 and ssl

DocImpact

Change-Id: I9aa1db6c9563868bf68ced7c4311daa870216ced
Merge "Upgrade WebOb to 1.2.3"
Relax various version constraints.

The current 3rd-party module requirements for cinder are still from the
nova git import. Usually, having an upper bound for requirements makes
only sense where API-incompatible changes are expected (like WebOb). For
the others it is better the only require a minimum version because
usually distributions differ on the shipped versions of these Python
modules. For instance, openSUSE has newer versions for almost all of
those and running the testsuite there works flawlessly.

Change-Id: I4614a2ea87084f927a557b9fb85dcd10ae9dfeff
support a configurable volume wiping method

Add a clear_volume() method that is called from delete_method().
This is only implemented at present for the base LVM driver.
This new clear_volume() method now supports these config vars:

  volume_clear = none, zero, shred
  volume_clear_size = size_in_MiB

'zero' is the default method and unchanged from previously.
'none' is used if security isn't a concern, or there is
  independent volume scrubbing.
'shred' is for more security conscious situations where
an overwrite count is required. Currently this defaults to 3 passes.

size_in_MiB can be used to limit the cleared area to the first
part of the volume, which can greatly speed up the operation,
and can be useful with encrypted volumes for example where
overwritting the encryption keys at the start is sufficient.

Fixes bug: 1022511
Change-Id: I7312678ff5d66156578501483a1b74e2a69e19a0
Merge "Enhance wsgi to listen on ipv6 address"
Merge "Support for SSL in wsgi.Server"
Merge "Update SolidFire Volume driver"
Merge "support a configurable volume wiping method"
Snapshot support for XenAPINFS

Related to blueprint xenapinfs-snapshots

Add support for snapshots, by implementing it with deep copies. Each
snapshot is a copy of the volume. Snapshot object has a new,
provider_location key, to store volume parameters.

Change-Id: Iea06246319ce82d5fcfe417419156ec83ef0a895
Move iscsi flags back to driver.py

During the lvm class create the iscsi flags were moved.
This is NOT correct, the flags belong with the ISCSI class
and should have been left in driver.py where ISCSIDriver
is defined.

Moving these breaks the inheritance structure for all those that
inherit from ISCSIDriver.

Change-Id: I1fda9158d9f8f01005875b87265d2dd87bcbc67f
Merge "Relax various version constraints."
Merge "Move iscsi flags back to driver.py"