Allow for only iterating over the most 'fresh' jobs

Instead of allowing the backend to determine how fresh
are the jobs yielded back (in the zookeeper case the jobs
are async populated so freshness depends on the freshness of
the watches established) allow the iteration method to support
a way to request (if backend supported) that the jobs iterated
be as fresh as possible.

Change-Id: I8a7c9d7d086ad5fa85dc842fb36e3356f781f057
Allow transient values to be stored in storage

Instead of requiring all values to be saved to storage
in a persistent manner allow certain values to be saved
in memory only (aka transient values) so that tasks can
accept things which should not be persisted (resources,
passwords, file handles, sockets...)

Breaking change: disallowing empty names for atom details
when they are ensured for (previously this was not done
at the storage level) but at the execution level. This
adds it in both places.

Change-Id: Ie68abdcf94f33fba5dbc05f03397aea5d82ea88f
Reuse already defined variable

The REVERTING variable is already defined as flow state. Reuse
it for the task state.

Change-Id: Ied67118be79e3ac933d5b7006720b72f8dc441b4
Get persistence backend via kwargs instead of conf

The persistence backend that the zookeeper backend uses to
load logbooks when they appear should come in via a kwarg
instead of coming in via conf (since it's not configuration).

Change-Id: I300d7681e5884a989549c8b84d29f04e9ea3d060
Add docs for jobs and jobboards

Add initial docs explaining the jobboard and job
concept and basic examples for how to use this
paradigm.

Part of blueprint jobboard-example-code

Change-Id: Ia8bc48967e55df6107dba272ae7187b18ba2a16e
Merge "Get persistence backend via kwargs instead of conf"
Merge "Reuse already defined variable"
Merge "Add docs for jobs and jobboards"
Ensure example abandons job when it fails

If a worker can't finish a job and the worker has not
crashed it should make sure that it manually abandons
the job that it claimed (if it crashes this release will
happen automatically).

Change-Id: I2d6894281915fd335bffa34c006aed023e88e9fc
Be better at failure tolerance

Instead of raising exceptions from a graph action
while tasks are still running or if a set of tasks
was scheduled and the last task could not be
scheduled (for example) just add the failure to the
failure list and continue processing what is actively
running.

This ensures that we don't leave futures in an
abandoned state and clean up our executing work which
is what all good behaving software should try to do.

Change-Id: I9256d7afa86f922273586a98b27442b6ba87767c
Stings -> Strings

Fix spelling mistake.

Change-Id: Iadbd3663a8f1cc8e95a754a165ef682490736256
Updated from global requirements

Change-Id: I4cb4b627a4cecce6c6768357486712711d350d5a
Add persistence docs + adjustments

- Increase the level of docs around why the persistence
  objects exist and what they are used for and how an
  engine interacts with persistence objects to accomplish
  the wider goals.
- Pass backend conf as arg, not kwarg and to match with the
  other fetch/load functions allow for providing a kwargs
  which can be used for backend specific params.

Change-Id: Ia83e63196bcfcc966f68212f84f79a1e9f18e8bc
Add a directed graph type (new types module)

Most of the utility graph functions we have can
be connected to a directed graph class that itself
derives (and adds on to) the networkx base class.

Doing this allows for functionality that isn't exposed
in networkx to be exposed in our subclass (which is a
useful pattern to have).

It also makes it possible (if ever needed) to replace
the networkx usage in taskflow with something else if
this ever becomes a major request.

Change-Id: I0a825d5637236d7b5dbdbda0d426adb0183d5ba3
Merge "Updated from global requirements"
Merge "Add persistence docs + adjustments"
Merge "Add a directed graph type (new types module)"
Merge "Allow for only iterating over the most 'fresh' jobs"
Merge "Be better at failure tolerance"
Merge "Allow transient values to be stored in storage"