Add a conductor considerations section

Add a small section in the conductor docs about
the cycling issue and give some resolutions that
can be applied as well as link to the better solution
which is garbage collection for jobs that are not
working out.

Also includes some tiny tweaks to other docs.

Change-Id: I73e9f8f5a8888eaf967d62723f6ffb45b02887c9
Add the arch/big picture omnigraffle diagram

To avoid losing this diagram again let's keep
it in the source tree so that others can modify
it and extend it as they desire (and also prevents
it from disappearing).

Will need omnigraffle to view:

https://www.omnigroup.com/omnigraffle

Change-Id: I44eb15722c9d6f2c63d965ff8f849efbdf15bbfb
Make greenexecutor not keep greenthreads active

Instead of keeping all greenthreads active even
if they are not being used only keep the greenthreads
active if there exists active work to complete.

This saves resources (each greenthread takes up
memory) and allows a pool to shrink and grow in
a more dynamic fashion.

Fixes bug 1339406

Change-Id: Idc8ab8447045915a0ffbaf21fa5c4bdb7a9e3593
Use checked commit when committing kazoo transactions

To avoid the case where the transaction appears to have
committed successfully (when it actually has not) create
a new checked_commit utility function and use it in the
zookeeper persistence backend to ensure that the transaction
has completed correctly (when it has not raise a new kazoo
exception that contains the failures to the caller, which can
then reraise that exception, examine the failures or do some
other action).

Change-Id: Ib169b36abb568a0e1516c4c786fccb30856661e1
Add basic WBE validation sanity tests

Add some basic tests that verify that our
new message format validation is working as
expected.

Part of blueprint wbe-message-validation

Change-Id: I64821407e52977a0e0ae0dff09f4b440717290cb
Use a check + create transaction when claiming a job

To avoid the case where a secondary entity is able to
consume and remove a job (and its associated lock) while
another entity is in the process of claiming a job by
getting that same lock we want to avoid this by having a
transaction be used that will enforce the job still exists
before the lock is created.

This ensures that a lock is not created on a job that does
not exist in the first place.

Fixes bug 1343029

Change-Id: I6c45cbb9dfb3a479658f4e605d37a6d3ac04d1b8
Use checked_commit() around consume() and abandon()

To ensure we reliably handle when a transaction fails
we should use the checked_commit() helper function instead
of the currently not fully exception handling kazoo
transaction commit function.

This should be addressed in the future with:

- https://github.com/python-zk/kazoo/pull/224
- https://github.com/python-zk/kazoo/pull/225

Those have not merged yet (or been released) so we need
to use a similar function in the meantime.

Change-Id: Icf83b7d4955c11227e733287170a7bd3ab372bd2
Add a pformat() failure method and use it in the conductor

When the conductors engine raises a wrapped failure (which may
contain a single failure from a remote worker, or may contain many
failures if there were more than one such failures...) it is very
useful to show more than just the basic exception information (or at
least provide methods that can print out the information in a nice
manner).

This commit adds a pformat() failure method that formats the failure
in a nice and understandable manner and then uses it in the conductor
dispatching routine so that the conductors LOGs are much more useful.

Change-Id: I46afacf54c9b4cae885c76c09b51e61d71fe623a
Update oslo-incubator to 037dee004c3e2239

This brought in/includes the following changes:

ad248f6658d5 Specify namedtuple_as_object=False when using simplejson
3d90045d2d Backport code for i18n to check lazy at runtime

Change-Id: I6ffb317f7e8fa57d79a7d0c6a9fc2517bc169e4b
Split requirements into py2 and py3 files

Since PBR now supports distinguishing between
a requirements file for py2 and a different requirements
file for py3 we should add the support for that separation
so that it is easier to package taskflow.

Change-Id: I3c5faa99a529bcb28aa0276b8d345a26f94b5af3
Remove sphinx examples emphasize-lines

The emphasize-lines was becoming inconsistent and
is hard to keep up to date in the correct manner so
removing it since we already provide introduction
comments in the examples and emphasizing/highlighting
them does not add much meaningful benefit.

Change-Id: I406dd606df75e7523703812803255418ec2f47d8
Allow worker count to be specified when no executor provided

When a multi-threaded engine is used it is nice to be able
to inform the engine how many workers should be created when
it creates its own executor. To allow this to be possible accept
a new keywork argument that can be used to set this value.

Change-Id: I0095d548249372440abbcf9b5c3b8fa841ca0ea9
Merge "Add a pformat() failure method and use it in the conductor"
Make the WBE worker banner information more meaningful

Add in more details that are displayed in the LOG when
a WBE worker is started up that is useful to show to be
able to help in debugging, or other informational and
operational purposes.

Example of the new output is the following:

TaskFlow v0.3.21.62 WBE worker.
Connection details:
  Driver = py-amqp v1.4.5
  Exchange = test
  Topic = test
  Transport = amqp
  Uri = amqp://guest@localhost:5672//
Powered by:
  Executor = concurrent.futures.thread.ThreadPoolExecutor
  Thread count = 3
Supported endpoints:
  - taskflow.tests.utils.NastyTask
  ...
  - taskflow.tests.utils.TaskMultiArgOneReturn
System details:
  Hostname = lappy.gateway.net
  Pid = 28364
  Platform = Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty
  Python = 2.7.6 (default, Mar 22 2014, 22:59:56)
  Thread id = 139875992315712

Change-Id: I6d7dba3406007ddc80cce96cfdbbfd25935a12ae
Updated from global requirements

Change-Id: I4758cb3c316356299595ca895e13cfcfc0b88123
Merge "Update oslo-incubator to 037dee004c3e2239"
Merge "Make greenexecutor not keep greenthreads active"
Updated from global requirements

Change-Id: Ifeed3788115e8fca7f9a23c293f7d91118743824
Merge "Use checked commit when committing kazoo transactions"
Merge "Updated from global requirements"