Add service mgmt extension.

Add capability to list, enable and disable cinder services on a node.

Change-Id: I6728fee46d7b15a009bc6c329414ede8c01852b4
Adding marker, pagination, sort key and sort direction to v2 api

Taking a cue from quantum, use glance's pagination function.

bp api-pagination

Change-Id: Ida7ade8d9332c88679849f7c640651df7e855abb
Adds synchronization to attach volume.

This patch adds the following changes:
1. Move db update for volume status during attach to volume.manager
2. Add lock synchronization to volume attach operation
3. Related unit tests

Fixes Bug 1087253
Change-Id: I15242766bf4cfa4da67789c485fdf6886983eb45
Add pyflakes

Add tox option to use pyflakes.

Change-Id: Ic77ec3f54cb89341d5b01feb3417e07f43d20c9c
Signed-off-by: Chuck Short <chuck.short@canonical.com>
Implement ability to Clone volumes in Cinder.

This implements the capability to create usable volume clones in Cinder,
for the LVM case we create a temporary snapshot to copy from so that
volumes can remain attached during cloning.  This works by passing
in a source-volume-id to the create command (similar to create-from-snapshot).

Currently we limit clone to the same Cinder node, and only for the base LVM driver.
All other drivers should raise NotImplemented, most inherit from the SANISCSIDriver,
so move the function there and raise until we have a general implementation
for SANISCSI based drivers.

Those drivers that inherit from ISCSI directly instead of SANISCSI,
add the function explicitly and raise NotImplementedError there as well.

Implements blueprint add-cloning-support-to-cinder

Change-Id: I72bf90baf22bec2d4806d00e2b827a594ed213f4
use deleted = False, not 0 for update

Fixes bug #1095401

When running tempest on cinder on postgresql we get a number
of errors because this call was trying to set deleted = 0.
On mysql 0 == False, but on postgresql this is a type error.

Change-Id: Ice85550e8138e0656c93a8a800afdbc5935e4bcc
Merge "Adds synchronization to attach volume."
Merge "Add pyflakes"
Merge "Adding marker, pagination, sort key and sort direction to v2 api"
Add missing library

According to pyflakes:

cinder/api/common.py:246: undefined name 'minidom'
cinder/api/common.py:254: undefined name 'minidom'
cinder/api/common.py:272: undefined name 'minidom'
cinder/api/common.py:284: undefined name 'minidom'

Add from xml.dom import minidom

Change-Id: Iadb1586ac15c29bb1f2154324e529976f82c19f0
Signed-off-by: Chuck Short <chuck.short@canonical.com>
Fix cmds clearing in TargetAdminTestCase.

clear_cmds() should reference self.cmds rather than cmds.

Change-Id: I1d191aee6a713cf9f2b08c46bf47c1959235cc15
Make WebOb version specification more flexible

Change the WebOb version to >=1.0.8 as an temporary
measure to allow the actual version update to 1.2.3
to roll out across the projects one at a time without
breaking the integration tests.

Change-Id: I9969b02a6896f7b952301e40e96f0ef6df8f34c0
Signed-off-by: Doug Hellmann <doug.hellmann@dreamhost.com>
Upgrade WebOb to 1.2.3

The version of WebOb being used in OpenStack was more than
1 year old. This change updates to the latest stable release.

Upgrading WebOb resolves a version conflict between OpenStack
and Pecan, the web framework used by the Ceilometer team for
version 2 of the ceilometer API.

Refer to http://docs.webob.org/en/latest/news.html
for the list of changes between 1.0.8 and 1.2.3.

bug 1092227

Change-Id: I44fdd11acbf1d5245e0c9c5a6199fd53200928ed
Signed-off-by: Doug Hellmann <doug.hellmann@dreamhost.com>
Use tempdir for lock_path in tests.

Commit: 2331d3336a6adf4fc13a3b187e91a5d1b1f7c723
introduced use of lockutils but wasn't setting a lock_path, in the tests.
As a result we were polluting cinder/openstack/ with cinder-xxxx lockfiles.

This patch adds a tempfile and cleanup to the test so we don't leave any
junk behind.

Change-Id: Idd9fee928ff0fcf3f8d1c30ee88c5306b88e290c
Merge "Make WebOb version specification more flexible"
Merge "Add service mgmt extension."
Merge "Implement ability to Clone volumes in Cinder."
Implement filter scheduler

In order to do more sophisticated scheduling (e.g. schedule based on volume
type), filter scheduler is introduced. Several changes are made to make this
possible, some of them are similar to the counterpart in Nova:

- add HostState class to host_manager in order to store volume capabilities
- implement get_volume_stats() method of iSCSIDriver as an example to
demonstrate how volume backend driver reports capabilities as well as status
- add scheduler_options.py and 'scheduler_json_config_location' flag to be
allow loading json configuration file for scheduler at run time
- port common filters/weights from oslo
- add capacity weigher (scheduler/weights/capacity.py) for picking up
target volume backend by weighing free capacity of the host. The default
behavior is to spread volumes across hosts; by changing the
'capacity_weight_multiplier' to negative number, volume placing behavior will
become stacking.
- add capacity filter which filters those hosts have insufficient storage space
to serve the request.
- add 'reserved_percentage' config option to indicate how much space is
reserved. Capacity reservation is needed when volume resize is enabled.
- change default scheduler to 'filter.FilterScheduler'.
- add 'publish_service_capabilities()' method to volume RPC API to allow
scheduler to have volume service report capabilities. This bumps volume RPC
API to version 1.2
- remove 'volume_force_update_capabilities' config option, volume status will be
report to scheduler in every checking.

The implication of this change to storage/backend driver developer:
- implementation of get_volume_stats() of the driver is now a *MUST*, filter
scheduler heavily relies on the status/capabilities reported by backend driver
to makeplacement decision.  To ensure Cinder works seamlessly on the storage
system, driver should at least report following capabilities/status:
----------------------+---------------------------+---------------------------
  Capability/Status   |      Description          |         Example
----------------------+---------------------------+---------------------------
 'volume_backend_name'| back-end name, string     | 'Example_Storage_Backend'
----------------------+---------------------------+---------------------------
  'vendor_name'       | vendor name, string       | 'OpenStackCinder'
----------------------+---------------------------+---------------------------
  'driver_version'    | version, string           |  '1.0a'
----------------------+---------------------------+---------------------------
  'storage_protocol'  | supported protocols,      | 'iSCSI', 'RBD', 'FC', 'NFS'
                      | string or list of strings | ['iSCSI', 'NFS', 'FC']
----------------------+---------------------------+---------------------------
  'total_capacity_gb' | capacity in GB, integer   |  102400
----------------------+---------------------------+---------------------------
  'free_capacity_gb'  | available capacity in GB, |  1000
                      | integer                   |
----------------------+---------------------------+---------------------------
'reserved_percentage' | reserved space in         |  0, 10
                      | percentage, integer       |
----------------------+---------------------------+---------------------------

The implication of this change to Cinder administrator:
- the default setting for filter scheduler should work well with the benefits
of:
  * being able to fully utilize capacity of backends (driver now has to report
  actul total space and space utilization and scheduler uses these info) not
  limited by the 'max_gigabytes' config option any more;
  * being able to choose placement policy between spreading & stacking for
  volume creation (by modifying the 'capacity_weight_multiplier' in
  CapacityWeigher)
- with filter scheduler, Cinder is now able to utilize the advanced features/
capabilities provided by different storage back-ends via: defining different
volume types with proper extra_specs. Volume types can be considered as sets
of back-end capabilities requirement.
 For example, a volume type which has 'storage_protocol':'FC' key/value pair
definition in its extra_spec can only be served by those back-ends who report
they support FiberChannel protocol. Another example is volume type has 'QoS'
requirement can only be served by back-ends support QoS.

Note/TODO:
* Currently scheduler makes its decision based on the status and capabilities
information reported by volume nodes, and these information is stored in memory
of scheduler process. More sophisticated way may be add on table in DB to
record status/capabilities of all volume nodes, like Nova does for compute nodes.

implement bp volume-type-scheduler

DocImpact

Change-Id: I8b5305bce3b24bcc85365037356a8f36d0107133
Add option to make exception format errors fatal.

Adds a new fatal_exception_format_errors config option which
defaults to False. This option is use to control how the base
CinderException class handles errors which can occur when it
formats error messages.

The motivation for this change is to be able to enable exception
format checking in our tests by setting fatal_exception_format_errors=True.

Change-Id: I2df952558d3f30a557cbb094f8bddcc37af9b5aa
Update exceptions to pass correct kwargs.

Updates a variety of Cinder exceptions so that they pass the correct
kwargs for proper exception message formatting. Previously
these exceptions would work but have incorrect error messages
because of how CinderException handled exception formatting errors.

Change-Id: Id1d15b112cad46a698227fa172d81a9ab586e66e