Add a conductor considerations section

Add a small section in the conductor docs about
the cycling issue and give some resolutions that
can be applied as well as link to the better solution
which is garbage collection for jobs that are not
working out.

Also includes some tiny tweaks to other docs.

Change-Id: I73e9f8f5a8888eaf967d62723f6ffb45b02887c9
Add the arch/big picture omnigraffle diagram

To avoid losing this diagram again let's keep
it in the source tree so that others can modify
it and extend it as they desire (and also prevents
it from disappearing).

Will need omnigraffle to view:

https://www.omnigroup.com/omnigraffle

Change-Id: I44eb15722c9d6f2c63d965ff8f849efbdf15bbfb
Make greenexecutor not keep greenthreads active

Instead of keeping all greenthreads active even
if they are not being used only keep the greenthreads
active if there exists active work to complete.

This saves resources (each greenthread takes up
memory) and allows a pool to shrink and grow in
a more dynamic fashion.

Fixes bug 1339406

Change-Id: Idc8ab8447045915a0ffbaf21fa5c4bdb7a9e3593
Use checked commit when committing kazoo transactions

To avoid the case where the transaction appears to have
committed successfully (when it actually has not) create
a new checked_commit utility function and use it in the
zookeeper persistence backend to ensure that the transaction
has completed correctly (when it has not raise a new kazoo
exception that contains the failures to the caller, which can
then reraise that exception, examine the failures or do some
other action).

Change-Id: Ib169b36abb568a0e1516c4c786fccb30856661e1
Add basic WBE validation sanity tests

Add some basic tests that verify that our
new message format validation is working as
expected.

Part of blueprint wbe-message-validation

Change-Id: I64821407e52977a0e0ae0dff09f4b440717290cb
Use a check + create transaction when claiming a job

To avoid the case where a secondary entity is able to
consume and remove a job (and its associated lock) while
another entity is in the process of claiming a job by
getting that same lock we want to avoid this by having a
transaction be used that will enforce the job still exists
before the lock is created.

This ensures that a lock is not created on a job that does
not exist in the first place.

Fixes bug 1343029

Change-Id: I6c45cbb9dfb3a479658f4e605d37a6d3ac04d1b8
Use checked_commit() around consume() and abandon()

To ensure we reliably handle when a transaction fails
we should use the checked_commit() helper function instead
of the currently not fully exception handling kazoo
transaction commit function.

This should be addressed in the future with:

- https://github.com/python-zk/kazoo/pull/224
- https://github.com/python-zk/kazoo/pull/225

Those have not merged yet (or been released) so we need
to use a similar function in the meantime.

Change-Id: Icf83b7d4955c11227e733287170a7bd3ab372bd2
Add a pformat() failure method and use it in the conductor

When the conductors engine raises a wrapped failure (which may
contain a single failure from a remote worker, or may contain many
failures if there were more than one such failures...) it is very
useful to show more than just the basic exception information (or at
least provide methods that can print out the information in a nice
manner).

This commit adds a pformat() failure method that formats the failure
in a nice and understandable manner and then uses it in the conductor
dispatching routine so that the conductors LOGs are much more useful.

Change-Id: I46afacf54c9b4cae885c76c09b51e61d71fe623a
LOG which requeue filter callback failed

When a requeue filter fails being called instead of just
logging the exception and traceback also log the callback
index which failed and the string representation of the
callback that failed so that when/if it is later examined to
determine the root cause these pieces of output can be used
to isolate the failing callback.

Also includes a drive-by commit to update the add requeue
filter callback documentation to explain what the callback
will be provided and expected to return.

Change-Id: Idf1ae63cfa706551ac0361e802f3b082783e0b9d
Allow WBE request transition timeout to be dynamic

To enable longer (or shorter) timeouts for a WBE submitted
request to transition out of the (PENDING, WAITING) states
allow the transition timeout that was previously set to 60
seconds to be provided as a WBE configuration option (it
still defaults to the previously fixed 60 seconds when it
is not provided).

Fixes bug 1356002

Change-Id: Idf384217004a334df03e2fff9150309fdfe08005
Remove db locks and use random db names for tests

Removed database locks against the database openstack_citest and
implemented random name generation for new databases created just
for persistence testing.
Once the locks were removed, using 'template1' as the initial db
for postgres connections was problematic, because postgres refuses
to create databases when there are multiple connections to
'template1'.  Switched to using 'postgres' as an initial db to use.
Changed _reset_database to _init_db since we are always working
with a brand new database, and removed the 'drop database' before
a 'create database.
Added a _remove_db method to remove the database once testing was
finished.

Change-Id: Iaf1c101df9c268da48db7432bcbc0467f6486bcd
Closes-Bug: 1327469
Update oslo-incubator to 037dee004c3e2239

This brought in/includes the following changes:

ad248f6658d5 Specify namedtuple_as_object=False when using simplejson
3d90045d2d Backport code for i18n to check lazy at runtime

Change-Id: I6ffb317f7e8fa57d79a7d0c6a9fc2517bc169e4b
Split requirements into py2 and py3 files

Since PBR now supports distinguishing between
a requirements file for py2 and a different requirements
file for py3 we should add the support for that separation
so that it is easier to package taskflow.

Change-Id: I3c5faa99a529bcb28aa0276b8d345a26f94b5af3
Remove sphinx examples emphasize-lines

The emphasize-lines was becoming inconsistent and
is hard to keep up to date in the correct manner so
removing it since we already provide introduction
comments in the examples and emphasizing/highlighting
them does not add much meaningful benefit.

Change-Id: I406dd606df75e7523703812803255418ec2f47d8
Allow worker count to be specified when no executor provided

When a multi-threaded engine is used it is nice to be able
to inform the engine how many workers should be created when
it creates its own executor. To allow this to be possible accept
a new keywork argument that can be used to set this value.

Change-Id: I0095d548249372440abbcf9b5c3b8fa841ca0ea9
Expand documention on failures and wrapped failures types

To make it more clear why these objects and types exist and when
and why they are used and reraised increase the verbosity of the
comments and associated inline documentation on these two classes
to help explain there existence & usefulness.

Change-Id: I742bbcd69c71df80e6de7becc8aee153bf16b34b
Merge "Add a pformat() failure method and use it in the conductor"
Make the WBE worker banner information more meaningful

Add in more details that are displayed in the LOG when
a WBE worker is started up that is useful to show to be
able to help in debugging, or other informational and
operational purposes.

Example of the new output is the following:

TaskFlow v0.3.21.62 WBE worker.
Connection details:
  Driver = py-amqp v1.4.5
  Exchange = test
  Topic = test
  Transport = amqp
  Uri = amqp://guest@localhost:5672//
Powered by:
  Executor = concurrent.futures.thread.ThreadPoolExecutor
  Thread count = 3
Supported endpoints:
  - taskflow.tests.utils.NastyTask
  ...
  - taskflow.tests.utils.TaskMultiArgOneReturn
System details:
  Hostname = lappy.gateway.net
  Pid = 28364
  Platform = Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty
  Python = 2.7.6 (default, Mar 22 2014, 22:59:56)
  Thread id = 139875992315712

Change-Id: I6d7dba3406007ddc80cce96cfdbbfd25935a12ae
Updated from global requirements

Change-Id: I4758cb3c316356299595ca895e13cfcfc0b88123
Merge "Update oslo-incubator to 037dee004c3e2239"