Implement filter scheduler

In order to do more sophisticated scheduling (e.g. schedule based on volume
type), filter scheduler is introduced. Several changes are made to make this
possible, some of them are similar to the counterpart in Nova:

- add HostState class to host_manager in order to store volume capabilities
- implement get_volume_stats() method of iSCSIDriver as an example to
demonstrate how volume backend driver reports capabilities as well as status
- add scheduler_options.py and 'scheduler_json_config_location' flag to be
allow loading json configuration file for scheduler at run time
- port common filters/weights from oslo
- add capacity weigher (scheduler/weights/capacity.py) for picking up
target volume backend by weighing free capacity of the host. The default
behavior is to spread volumes across hosts; by changing the
'capacity_weight_multiplier' to negative number, volume placing behavior will
become stacking.
- add capacity filter which filters those hosts have insufficient storage space
to serve the request.
- add 'reserved_percentage' config option to indicate how much space is
reserved. Capacity reservation is needed when volume resize is enabled.
- change default scheduler to 'filter.FilterScheduler'.
- add 'publish_service_capabilities()' method to volume RPC API to allow
scheduler to have volume service report capabilities. This bumps volume RPC
API to version 1.2
- remove 'volume_force_update_capabilities' config option, volume status will be
report to scheduler in every checking.

The implication of this change to storage/backend driver developer:
- implementation of get_volume_stats() of the driver is now a *MUST*, filter
scheduler heavily relies on the status/capabilities reported by backend driver
to makeplacement decision.  To ensure Cinder works seamlessly on the storage
system, driver should at least report following capabilities/status:
----------------------+---------------------------+---------------------------
  Capability/Status   |      Description          |         Example
----------------------+---------------------------+---------------------------
 'volume_backend_name'| back-end name, string     | 'Example_Storage_Backend'
----------------------+---------------------------+---------------------------
  'vendor_name'       | vendor name, string       | 'OpenStackCinder'
----------------------+---------------------------+---------------------------
  'driver_version'    | version, string           |  '1.0a'
----------------------+---------------------------+---------------------------
  'storage_protocol'  | supported protocols,      | 'iSCSI', 'RBD', 'FC', 'NFS'
                      | string or list of strings | ['iSCSI', 'NFS', 'FC']
----------------------+---------------------------+---------------------------
  'total_capacity_gb' | capacity in GB, integer   |  102400
----------------------+---------------------------+---------------------------
  'free_capacity_gb'  | available capacity in GB, |  1000
                      | integer                   |
----------------------+---------------------------+---------------------------
'reserved_percentage' | reserved space in         |  0, 10
                      | percentage, integer       |
----------------------+---------------------------+---------------------------

The implication of this change to Cinder administrator:
- the default setting for filter scheduler should work well with the benefits
of:
  * being able to fully utilize capacity of backends (driver now has to report
  actul total space and space utilization and scheduler uses these info) not
  limited by the 'max_gigabytes' config option any more;
  * being able to choose placement policy between spreading & stacking for
  volume creation (by modifying the 'capacity_weight_multiplier' in
  CapacityWeigher)
- with filter scheduler, Cinder is now able to utilize the advanced features/
capabilities provided by different storage back-ends via: defining different
volume types with proper extra_specs. Volume types can be considered as sets
of back-end capabilities requirement.
 For example, a volume type which has 'storage_protocol':'FC' key/value pair
definition in its extra_spec can only be served by those back-ends who report
they support FiberChannel protocol. Another example is volume type has 'QoS'
requirement can only be served by back-ends support QoS.

Note/TODO:
* Currently scheduler makes its decision based on the status and capabilities
information reported by volume nodes, and these information is stored in memory
of scheduler process. More sophisticated way may be add on table in DB to
record status/capabilities of all volume nodes, like Nova does for compute nodes.

implement bp volume-type-scheduler

DocImpact

Change-Id: I8b5305bce3b24bcc85365037356a8f36d0107133
Merge "Implement filter scheduler"
Revert "Implement filter scheduler"

This reverts commit 8dd2140c4c03582965d74232c7b61aef34e43661
Merge "Revert "Implement filter scheduler""
Implement filter scheduler

In order to do more sophisticated scheduling (e.g. schedule based on volume
type), filter scheduler is introduced. Several changes are made to make this
possible, some of them are similar to the counterpart in Nova:

- add HostState class to host_manager in order to store volume capabilities
- implement get_volume_stats() method of iSCSIDriver as an example to
demonstrate how volume backend driver reports capabilities as well as status
- add scheduler_options.py and 'scheduler_json_config_location' flag to be
allow loading json configuration file for scheduler at run time
- port common filters/weights from oslo
- add capacity weigher (scheduler/weights/capacity.py) for picking up
target volume backend by weighing free capacity of the host. The default
behavior is to spread volumes across hosts; by changing the
'capacity_weight_multiplier' to negative number, volume placing behavior will
become stacking.
- add capacity filter which filters those hosts have insufficient storage space
to serve the request.
- add 'reserved_percentage' config option to indicate how much space is
reserved. Capacity reservation is needed when volume resize is enabled.
- add 'publish_service_capabilities()' method to volume RPC API to allow
scheduler to have volume service report capabilities. This bumps volume RPC
API to version 1.2
- remove 'volume_force_update_capabilities' config option, volume status will be
report to scheduler in every checking.

The implication of this change to storage/backend driver developer:
- implementation of get_volume_stats() of the driver is now a *MUST*, filter
scheduler heavily relies on the status/capabilities reported by backend driver
to makeplacement decision.  To ensure Cinder works seamlessly on the storage
system, driver should at least report following capabilities/status:
----------------------+---------------------------+---------------------------
  Capability/Status   |      Description          |         Example
----------------------+---------------------------+---------------------------
 'volume_backend_name'| back-end name, string     | 'Example_Storage_Backend'
----------------------+---------------------------+---------------------------
  'vendor_name'       | vendor name, string       | 'OpenStackCinder'
----------------------+---------------------------+---------------------------
  'driver_version'    | version, string           |  '1.0a'
----------------------+---------------------------+---------------------------
  'storage_protocol'  | supported protocols,      | 'iSCSI', 'RBD', 'FC', 'NFS'
                      | string or list of strings | ['iSCSI', 'NFS', 'FC']
----------------------+---------------------------+---------------------------
  'total_capacity_gb' | capacity in GB, integer   |  102400
----------------------+---------------------------+---------------------------
  'free_capacity_gb'  | available capacity in GB, |  1000
                      | integer                   |
----------------------+---------------------------+---------------------------
'reserved_percentage' | reserved space in         |  0, 10
                      | percentage, integer       |
----------------------+---------------------------+---------------------------

The implication of this change to Cinder administrator:
- the default setting for filter scheduler should work well with the benefits
of:
  * being able to fully utilize capacity of backends (driver now has to report
  actul total space and space utilization and scheduler uses these info) not
  limited by the 'max_gigabytes' config option any more;
  * being able to choose placement policy between spreading & stacking for
  volume creation (by modifying the 'capacity_weight_multiplier' in
  CapacityWeigher)
- with filter scheduler, Cinder is now able to utilize the advanced features/
capabilities provided by different storage back-ends via: defining different
volume types with proper extra_specs. Volume types can be considered as sets
of back-end capabilities requirement.
 For example, a volume type which has 'storage_protocol':'FC' key/value pair
definition in its extra_spec can only be served by those back-ends who report
they support FiberChannel protocol. Another example is volume type has 'QoS'
requirement can only be served by back-ends support QoS.

Note/TODO:
* Currently scheduler makes its decision based on the status and capabilities
information reported by volume nodes, and these information is stored in memory
of scheduler process. More sophisticated way may be add on table in DB to
record status/capabilities of all volume nodes, like Nova does for compute nodes.

implement bp volume-type-scheduler

DocImpact

Change-Id: I296b3727db8de0d4cf085fac602d122a7b474842
Snapshot support for XenAPINFS

Related to blueprint xenapinfs-snapshots

Add support for snapshots, by implementing it with deep copies. Each
snapshot is a copy of the volume. Snapshot object has a new,
provider_location key, to store volume parameters.

Change-Id: Iea06246319ce82d5fcfe417419156ec83ef0a895
Format code - eliminate long lines
Add cinder specific bits

Change-Id: I56a9d3e4bcc8833a37bbdbe51ed60c511a35a230
fix bad hacks

Change-Id: Ie79db5e0379ae535594ad79973a80ced348cd72b
Rename migration script

Change-Id: I7ac77249b1f37238e01bcfee7bd0393621827e59
Added tests

Change-Id: I15ed441d58043a3b6e2024209f5b566eeab637fe
Remove non-used imports, variables

Change-Id: I8cff795db0336676c818b960b7ac58459aa83639
Add relation and foreign key so snapshot can reference to its volume

Change-Id: I9d4393897feabacdedaf74f81019317baf1e4897
Always clean-up sr and pbds, even if an exception occured

Change-Id: I12bb91f8d6b7696afe5fde67652a61d6d8189548
Increment migrate version numbers

Change-Id: I03e69ca4a1f303670e9e331176ad362bba97c428
Proper sqlite downgrade script added

Change-Id: I8da8e83249ed2005a4268a175215df845249d79d
Modify copyright, and format

Change-Id: If6862aea648613c9c9dc19f60cccc3b506bc05fb
amend whitespaces

Change-Id: I728d5984f766c09ada2c3513e381aa9335445b2c