Add a conductor considerations section

Add a small section in the conductor docs about
the cycling issue and give some resolutions that
can be applied as well as link to the better solution
which is garbage collection for jobs that are not
working out.

Also includes some tiny tweaks to other docs.

Change-Id: I73e9f8f5a8888eaf967d62723f6ffb45b02887c9
Update oslo incubator code to commit 0b02fc0f36814968

gettextutils
 - de4adbc4 (pep8: fixed multiple violations)
importutils
 - 1173e46981a6d (Remove ValueError when accessing sys.modules)
jsonutils
 - 0d7296f6885 (Add kwargs to jsonutils.load(s) functions)
network_utils
 - 9502a21bfcc (Use the standard python logging in network_utils)
 - e433899db279743 (Enable configuring tcp keepalive)
 - 29a95c2ce11f9 (Avoid raising index error when no host)
strutils
 - cb5a804bd4 (Move `mask_password` to strutils)
timeutils
 - 250cd88c4040c1f9 (Fixed a new pep8 error and a small typo)

Change-Id: I8a4265127d1abf90f0a83e31723490bd0c25e8ae
Remove need to do special exception catching in parse_uri

The upstream oslo incubator code was fixed to not throw a
index error when parts of the uri are missing, so we can
remove our modification that handles these errors and just
rely on the fixed version.

Change-Id: I01c6a647858ca45aeb4685a68cb3e28b575e1e05
Add the arch/big picture omnigraffle diagram

To avoid losing this diagram again let's keep
it in the source tree so that others can modify
it and extend it as they desire (and also prevents
it from disappearing).

Will need omnigraffle to view:

https://www.omnigroup.com/omnigraffle

Change-Id: I44eb15722c9d6f2c63d965ff8f849efbdf15bbfb
Make greenexecutor not keep greenthreads active

Instead of keeping all greenthreads active even
if they are not being used only keep the greenthreads
active if there exists active work to complete.

This saves resources (each greenthread takes up
memory) and allows a pool to shrink and grow in
a more dynamic fashion.

Fixes bug 1339406

Change-Id: Idc8ab8447045915a0ffbaf21fa5c4bdb7a9e3593
Cleanup some of the example code & docs

This commit makes a set of small adjustments to examples.

- Rework some of the comments to be more clear.
- Add links to the original source tree file.
- Rename some of the examples to make it clear the
  concept the example is intented to show.
- Move some common example functionality to the
  example utility file.

Change-Id: I858e0dbf72fe8cb40a05bfdbb0857720ffb71c7f
Allow a jobs posted book to be none by default

Not all users of jobs and jobboards are using the
associated book that accompanies a job, instead they
are fine with just using the name and the details
that can be provided for usage in their application.

Example: http://review.openstack.org/#/c/91763/

To allow the optional usage of books with jobs (which
is already supported) by default set the book to none
and allow it to be provided on a as needed basis.

Change-Id: I69e370a733e44c45c62177008838c259fd9c9a7c
Use checked commit when committing kazoo transactions

To avoid the case where the transaction appears to have
committed successfully (when it actually has not) create
a new checked_commit utility function and use it in the
zookeeper persistence backend to ensure that the transaction
has completed correctly (when it has not raise a new kazoo
exception that contains the failures to the caller, which can
then reraise that exception, examine the failures or do some
other action).

Change-Id: Ib169b36abb568a0e1516c4c786fccb30856661e1
Allow handlers to provide validation callables

In order to reject messages before they are processed
make it possible to provide a handler pair that will
validate the incoming message and either reject or allow
the message to be passed to the provided handler.

Part of blueprint wbe-message-validation

Change-Id: Ibd6ee40020c6b98283f40d5bd59e8950d63b7f71
WBE notification message validation

Add send and receive validation of the notify
message that is sent between executors and workers
to be more robust around invalid message formats
being sent and received.

Part of blueprint wbe-message-validation

Change-Id: I7300d6f2d00e48c4f989c7f958a028bdff4afdd4
WBE response message validation

Add receive validation of the response message
sent by workers for progress updates and for
result completion to be more robust around invalid
message formats being sent and received.

Part of blueprint wbe-message-validation

Change-Id: I583b9d6344224724d147f2a5b10ddbd23d6a5fdc
WBE request message validation

Add receive validation of the request message
sent by executors to workers to initiate work
requests so that we can be more robust around
invalid message formats being sent and received.

Part of blueprint wbe-message-validation

Change-Id: If7fc4c870252b6ef29e626a874e42c82d3879512
Add basic WBE validation sanity tests

Add some basic tests that verify that our
new message format validation is working as
expected.

Part of blueprint wbe-message-validation

Change-Id: I64821407e52977a0e0ae0dff09f4b440717290cb
Improve WBE testing coverage

Test more of the message pump process and verify
that messages are received and sent correctly by
using a latch type and waiting for the desired
number of messages to be triggered.

Test the full server and executor pipeline by
setting up threads and a in-memory queue/exchange
that is used by these threads for task execution.

Change-Id: I62296f12eee6fa00559e84068ec5ee2a6d4bc0dc
Use a check + create transaction when claiming a job

To avoid the case where a secondary entity is able to
consume and remove a job (and its associated lock) while
another entity is in the process of claiming a job by
getting that same lock we want to avoid this by having a
transaction be used that will enforce the job still exists
before the lock is created.

This ensures that a lock is not created on a job that does
not exist in the first place.

Fixes bug 1343029

Change-Id: I6c45cbb9dfb3a479658f4e605d37a6d3ac04d1b8
Use checked_commit() around consume() and abandon()

To ensure we reliably handle when a transaction fails
we should use the checked_commit() helper function instead
of the currently not fully exception handling kazoo
transaction commit function.

This should be addressed in the future with:

- https://github.com/python-zk/kazoo/pull/224
- https://github.com/python-zk/kazoo/pull/225

Those have not merged yet (or been released) so we need
to use a similar function in the meantime.

Change-Id: Icf83b7d4955c11227e733287170a7bd3ab372bd2
add pre/post execute/retry callbacks to tasks

This enables us to execute code to set up or tear down
global state in running tasks.

Change-Id: Ib1e5d03ab46b3ce1d03fa83b91bf437fa950b758
Implements: blueprint task-callbacks
Merge "add pre/post execute/retry callbacks to tasks"
Merge "Allow handlers to provide validation callables"
Add a pformat() failure method and use it in the conductor

When the conductors engine raises a wrapped failure (which may
contain a single failure from a remote worker, or may contain many
failures if there were more than one such failures...) it is very
useful to show more than just the basic exception information (or at
least provide methods that can print out the information in a nice
manner).

This commit adds a pformat() failure method that formats the failure
in a nice and understandable manner and then uses it in the conductor
dispatching routine so that the conductors LOGs are much more useful.

Change-Id: I46afacf54c9b4cae885c76c09b51e61d71fe623a