Repository: DataBiosphere/toil

Issue:
Message Bus is confusing
It does not have enough architectural documentation about where the messages really live and how it can be used to do things.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1231)
┆friendlyId: TOIL-1231



Discussion:

Resolving patch:
diff --git a/src/toil/bus.py b/src/toil/bus.py
--- a/src/toil/bus.py
+++ b/src/toil/bus.py
@@ -14,6 +14,51 @@
 
 """
 Message types and message bus for leader component coordination.
+
+Historically, the Toil Leader has been organized around functions calling other
+functions to "handle" different things happening. Over time, it has become very
+brittle: exactly the right handling functions need to be called in exactly the
+right order, or it gets confused and does the wrong thing.
+
+The MessageBus is meant to let the leader avoid this by more losely coupling
+its components together, by having them communicate by sending messages instead
+of by calling functions.
+
+When events occur (like a job coming back from the batch system with a failed
+exit status), this will be translated into a message that will be sent to the
+bus. Then, all the leader components that need to react to this message in some
+way (by, say, decrementing the retry count) would listen for the relevant
+messages on the bus and react to them. If a new component needs to be added, it
+can be plugged into the message bus and receive and react to messages without
+interfering with existing components' ability to react to the same messages.
+
+Eventually, the different aspects of the Leader could become separate objects.
+
+By default, messages stay entirely within the Toil leader process, and are not
+persisted anywhere, not even in the JobStore.
+
+The Message Bus also provides an extension point: its messages can be
+serialized to a file by the leader (see the --writeMessages option), and they
+can then be decoded using MessageBus.scan_bus_messages() (as is done in the
+Toil WES server backend). By replaying the messages and tracking their effects
+on job state, you can get an up-to-date view of the state of the jobs in a
+workflow. This includes information, such as whether jobs are issued or
+running, or what jobs have completely finished, which is not persisted in the
+JobStore.
+
+The MessageBus instance for the leader process is owned by the Toil leader, but
+the BatchSystem has an opportunity to connect to it, and can send (or listen
+for) messages. Right now the BatchSystem deos not *have* to send or receive any
+messages; the Leader is responsible for polling it via the BatchSystem API and
+generating the events. But a BatchSystem implementation *may* send additional
+events (like JobAnnotationMessage).
+
+Currently, the MessageBus is implemented using pypubsub, and so messages are
+meant to be both generated by and handled in a single Thread, the Toil leader's
+main loop thread. If other components send events, they need to do it when
+called into by that thread, not from their own daemon threads. Communication
+between processes is allowed using MessageBus.connect_output_file() and
+MessageBus.scan_bus_messages().
 """
 
 import collections


Test patch: