Add ability to compare two verification results

Currently there is no built-in way in Rally to see the differences
between two verification runs. This feature adds a compare sub-command
to rally verify that takes two verification IDs as input and output the
differences between two verifications in the desired format, html, csv,
json or pprint. Output may be printed to std out or directed to file
destination specified.

Tasks
- Add compare sub-command to rally verify.
- Add ability to specify output format and file name of comparison
  report.
  Initial support will allow for output to either HTML, pretty print
  or CSV.
- Create rally/verification/verifiers/tempest/diff.py that handles all
  the comparing and output of results.
- Create required mako template (compare.mako) and renderer
  (compare2html.py) class for HTML output.
- Added percentage threshold argument for determining execution time
  differences

Change-Id: I0276e431d7d1ad89b1b28306cd0e0ea7e1174217
Implements: blueprint tempest-compare
Sahara Cluster error state handling

If any Sahara cluster gets into Error state in benchmarks or context
the exception will be raised.

Sahara Cluster context is refactored to bring the wait method to the
class level.

Change-Id: Id437babd6d9e06c99481f2f62c02c2e95570cf81
Updated from global requirements

Change-Id: I5840fce83406973cb45d95a667256eb1472ad7ba
Merge "Add ability to compare two verification results"
Removing redundancy in (a==b) and 1 or 0

Change-Id: I95ac07b1199dc2282b7a238e6fe44477885226c3
Add SLA to Designate scenarios

Change-Id: I175c0fd5a2d171825ec59cb1b099953345306953
Refactor generic cleanup mechanism part 1

Current cleanup mechanism is awful. It's hardcoded, without repeat on failures,
contains a lot of mistakes, in some places it is too fast (in case of deletion of VMs)
in other to slow (like deletion of users). As well there is mix of cleanup mechanism
and resource cleanup mechanism.
To resolve these all issues this patch introcude a cleanup engine that resolves all issues
above.
It's resource based, so to add new resouce you should just a make subclass of base.Resource
and probably override some of methods like (list, delete, is_deleted) and that's all.
All complexity of managing:
0) waiting until async deletion is finished
1) repeat on failure logic
2) greaceful failure handling
3) parallelization
4) plugin support
Is hidden deep inside cleanup engine
+ bonus we are able to specify now single resource (without clenaping whole service)

PART 1:
-------
* Add new generic cleanup engine
* Fix bug in broker (queue can be empty because of multithreading access)
* Add retry method in rally utils, allow as to repeat few times method on failures

bp benchmark-context-cleanup-refactor

Change-Id: I9f6fb0642b73b7b4d3f6f14de4ef6943a4fb7320
Fix tempest.conf subnets configuration
 
It will run into exception of 'KeyError': 0, 
because it should get 'subnets' first from the dictionary.

Change-Id: I74ca5ab53610e28d3933bad8a24c4272c51f92c4
Refactor UserGenerator

1. In the current implementation, it is possible for the UserGenerator to
   create a tenant in _create_tenant_users() and then fail at user creation,
   without returning the created tenant data. As a result, created tenants don't
   get deleted during cleanup.

   Here we refactor the tenants / users creation code by splitting it to two
   different methods, both launched via the Broker pattern.

   We remove the run_concurrent() method that is no longer used.

2. To fix a security issue, we change the password used for temporary users from
   "password" to a randomly generated uuid.

3. We rename the "concurrent" config option in the user context to
   "resource_management_workers".

Change-Id: I0b7f6b5677bb564c044c93aa1a9aeafc3b9f53cc
Closes-Bug: #1347215
Closes-Bug: #1365758
Merge "Refactor UserGenerator"
Merge "Refactor generic cleanup mechanism part 1"
Refactor generic cleanup mechanism part 2

Current cleanup mechanism is awful. It's hardcoded, without repeat on failures,
contains a lot of mistakes, in some places it is too fast (in case of deletion of VMs)
in other to slow (like deletion of users). As well there is mix of cleanup mechanism
and resource cleanup mechanism.
To resolve these all issues this patch introcude a cleanup engine that resolves all issues
above.
It's resource based, so to add new resouce you should just a make subclass of base.Resource
and probably override some of methods like (list, delete, is_deleted) and that's all.
All complexity of managing:
0) waiting until async deletion is finished
1) repeat on failure logic
2) greaceful failure handling
3) parallelization
4) plugin support
Is hidden deep inside cleanup engine
+ bonus we are able to specify now single resource (without clenaping whole service)

PART 2:
-------

* Write cleanup plugins for all tracked resources (on currnet moment)
* Switch cleanap context to use cleanup engine isntead of ald cleanup.utils
* Merge admin and user cleanup context (cause they are quite short now)
* Change quotas scenairos cleanup context configuration
* Fix resource names that are created by ceilometer scenarios

bp benchmark-context-cleanup-refactor

Change-Id: I85ab881fd65da4bb8263b4f72b9f751673ae5099
Updated from global requirements

Change-Id: Iba8167336fc991a85c52902ea0fa5cce6b8d00c8
Merge "Sahara Cluster error state handling"
Merge "Updated from global requirements"
Refactor generic cleanup mechanism part 3

Current cleanup mechanism is awful. It's hardcoded, without repeat on failures,
contains a lot of mistakes, in some places it is too fast (in case of deletion of VMs)
in other to slow (like deletion of users). As well there is mix of cleanup mechanism
and resource cleanup mechanism.
To resolve these all issues this patch introcude a cleanup engine that resolves all issues
above.
It's resource based, so to add new resouce you should just a make subclass of base.Resource
and probably override some of methods like (list, delete, is_deleted) and that's all.
All complexity of managing:
0) waiting until async deletion is finished
1) repeat on failure logic
2) greaceful failure handling
3) parallelization
4) plugin support
Is hidden deep inside cleanup engine
+ bonus we are able to specify now single resource (without clenaping whole service)

PART 3:
-------

*) Refactor all contexts cleanups method to use new generic cleanup engine
   insted of cleanup.utils
*) Remove obsolate cleanup.utils
*) Fix all tests

bp benchmark-context-cleanup-refactor

Change-Id: I70557e6ebb56bbe565792d9ee854d3e78428a881
Fix typo in Ceilometer's class in the cleanup

Ceilometer was written as Celiometer. This change
corrects that typo.

Change-Id: Iced23b806fc022ac4cf0871efb9f85c6ed197606
Merge "Add SLA to Designate scenarios"
Merge "Refactor generic cleanup mechanism part 3"
Updated from global requirements

Change-Id: Ib5194aad4fe4c50f1499438b0959d1458fa6e3e3