Have the dispatch_job function return a future

To make it easier to add in a multi-threaded conductor
convert the base dispatch_job function to return a future
object. This future object will contain a single result,
whether the job should be consumed or abandoned. In the
single threaded conductor its dispatch_job function will
return a future, after completing the job (in a multi
threaded conductor it would not return a future after
doing the work).

Change-Id: I077334820d36c64e272e93d158e3a0cd0d66a937
Be smarter about required flow symbols

Instead of blindly assuming all the symbols that
are provided automatically work for all flows even
if the flow has ordering constraints we should set
the base flow class requires property to be abstract
and provide flow specific properties that can do the
appropriate analysis to determine what the flows
unsatisfied symbol requirements actually are.

Part of blueprint taskflow-improved-scoping

Change-Id: Ie149c05b3305c5bfff9d9f2c05e7e064c3a6d0c7
Revamp the symbol lookup mechanism

To complement the future changes in patterns we also want
to allow the execution of patterns to be affected in a similar
manner so that symbol lookup is no longer as confined as it was.

This change adds in the following:

- Symbol lookup by walking through an atoms contained scope/s.
- Better error messaging when symbols are not found.
- Adjusted & new tests (existing ones work).
- Better logging of the symbol lookup mechanism (helpful
  during debugging, although it is very verbose...)

Part of blueprint taskflow-improved-scoping

Change-Id: Id921a4abd9bf2b7b5c5a762337f8e90e8f1fe194
Relax the linear flow symbol constraints

In order to make it possible to have a symbol
tree we need to relax and remove the constraints
that are being imposed by the linear constraints
and later move those constraint checks and validations
into the engines compilation stage.

Part of blueprint taskflow-improved-scoping

Change-Id: I6efdc821ff991e83572d89f56be5c678d007f9f8
Relax the unordered flow symbol constraints

In order to make it possible to have a symbol
tree we need to relax and remove the constraints
that are being imposed by the unordered constraints
and later move those constraint checks and validations
into the engines compilation stage.

Part of blueprint taskflow-improved-scoping

Change-Id: I80718b4bc01fbf0dce6a95cd2fac7e6e2e1814d1
Relax the graph flow symbol constraints

In order to make it possible to have a symbol
tree we need to relax the constraints that are
being imposed by the graph flow.

Part of blueprint taskflow-improved-scoping

Change-Id: I2e14de2131c3ba4e3e4eb3108477583d0f02dae2
Add a couple of scope shadowing test cases

Since we now support symbol name shadowing we
should add a few test cases that ensure that
the correct shadowing order/capability is working
as expected.

Change-Id: I95bad23a33304ddbfa8715a00c913891d4c51f5d
Add a state machine copy() method

In order to move the states.py constants to a state machine
object we need to be able to copy that object that will be
defined there so that it can be used by those states users.

Change-Id: I5fc21370d91ff578bd21c74f4bd7b8c0e130b144
Remove useless __exit__ return

By default we will return none, and that qualifies as
falsey, so we don't need to explicitly return false when
the default will do just fine.

Change-Id: I93a858177be138bf09abfc32897c96a991f69386
Add a timing listener that also prints the results

Instead of just recording them it can also be quite
useful (especially for debugging) to print the start
and stop timings as they occur.

Also adds an example that shows how this can be used
and an explanation of why it is useful to have this type
of capability.

Part of blueprint more-examples

Change-Id: Id2dc3f8dc9ac94e511470e39f499f325b33537ee
Link a few of the classes to implemented features/bugs in python

This adds comments that associate the classes we have for threading
usage to upstream bugs in python where similar features are being
created (and potentially supported upstream).

When we are able to reduce the number of supported python versions
we can/should try to remove our implementations and move to the ones
that may showup in the python standard library instead.

Change-Id: I7b58380aeb57a58fa3b3c424c9f39de30f44f0e9
Ensure state machine can be frozen

To match the other types ability to be frozen so that
they can no longer be mutated add a freeze() method to
the state machine type that ensures that subsequent
add_state, add_reaction, add_transition method calls will
raise an exception.

This is quite useful when the state machine is constructed
in one function and the creator wants to stop further adds
by other functions. To start use this freeze() capability in
the runner state machine when a machine build is requested.

Part of blueprint runner-state-machine

Change-Id: I61488e4158b38d39017435af008382f28d800049
Better handle the tree freeze method

Instead of dynamically replacing the existing method
with a new method, just have the existing method check
if the node has been frozen and immediately abort, this
workers better with decorators, subclassing...

This also unifies how freezing is done across all types
that support it, ensuring that the frozen attribute can
be set by users (if they so choose).

Change-Id: I1e6c6568b7f91765d654d25ca6e68e9b568603fc
Remove no longer needed r/w lock interface base class

This interface/class was put in place when this code was being
more actively developed to ensure that both the dummy and the
actual class had the same methods, since these two classes
have apis that are now stable we no longer need to have a base
class to enforce the implemented api that itself provides no
added functionality.

Change-Id: Ida4f86b308941ff708db5395a1d266c0c4b75815
Document more function/class/method params

To help fill out the docs start adding on docstrings
that document what a parameter is and what its meaning
and usage is to help users better understand the parameter.

Part of ongoing bug 1374202

Change-Id: I9f9a83cfb763a0a05d22efca4e0f80627ba8ca8f
Bring in a newer optional eventlet

Since the openstack requirements repo just accepted eventlet>=0.15.1
we might as well also use that version in our optional requirements
and remove a piece of code that was dealing with a bug that was fixed
in eventlet 0.15.

Change-Id: I9b4f9061c7adb7d72315315f41bb0d742b6f56b5
Increase robustness of WBE message and request processing

When a notification request/response can't be processed ensure
we log an error message at the same level as the other function
that sends back responses.

Also adds in a return boolean from the _reply message (which
is used for the X number of replies to a servers task request)
function and use this boolean to determine if the worker should
attempt to perform the final handler call that activates the
desired task.

Change-Id: I7f3914c126f39c56d0d2e3dfe02f3b112391ff43
Expose only `ensure_atom` from storage

Move the storage ensuring logic from being split
across the engine and the storage layer and expose
only a single `ensure_atom` function that does
the work instead.

This also removes the access to the `ensure_task`
and `ensure_retry` methods as the internals of the
`ensure_atom` function is now the only location
that needs to use these two functions.

This reduces the need to do type specific atom
checks in the non-storage components (which we
want to reduce overall).

Breaking change: removes the public methods named
`ensure_task` and `ensure_retry` (which should not
be used externally anyway) from the storage object
and makes those internal/private methods instead.

Change-Id: I3a0f1f0dd777a1633b4937e16b50030275c84d1d
Use timeutils functions instead of misc.wallclock

The common oslo timeutils functions can perform the
same time methods using the better datetime objects than
using the raw unix timestamps directly, so in order to
reduce a little bit of code just use the functions that
module provides instead of our own.

Also adds a few more tests that validate the various runtime
errors being thrown to ensure they are thrown when expected
and handles the case where time goes backwards (say when ntpd
updates) in a more reliable manner (by not becoming negative).

Change-Id: I6153ff8379833844105545ddb21dede65a7d4d3a
Add a more dynamic/useful logging listener

Both cinder and glance are starting to share the same logic
for there engine notification listener, so instead of having
them copy around that code it will be much nicer if taskflow
can just provide itself a more capable listener that both
can share and use directly.

This avoids users of taskflow having to understand more about
the internals of taskflow and its associated state then they
likely need to understand (which makes taskflow easier to use
and less work to integrate).

Relevant locations where this already exists:

- https://github.com/openstack/cinder/blob/master/cinder/flow_utils.py
- https://review.openstack.org/#/c/85211/

Change-Id: I98eeb180b31bd488ae0eadd730e1530d7bae1f1f