nathanmarz/storm — Issue #146: Tracer bullets
Author: velvia
Created: 2012-03-01 19:26:41+00:00
State: open

Debugging the flow of tuples through a Storm topology can be pretty tedious. One might have to do lots of logging and watch many log files, or do other kinds of instrumentation. It would be great to include a system to select certain tuples for tracing, and track the progress of those tuples through the topology.

Here is a use case:
* Suppose one were to do stats aggregation using Storm. Some things I might want to ensure are:
* Is the aggregation and flush happening in a timely way?
* Are there hotspots?
* Are there unexpected latencies? Are some bolts taking a long time?

To answer the above questions, I might select a random sample of tuples, or maybe a random sample of a specific subset of tuples. The tuples to be traced could be tagged with a special attribute.

I would want to track the following events:
* Spout emit - send (task id, spout name, timestamp)
* For each bolt:
* When a traced tuple arrives and execute() is called: (task id, bolt name, timestamp)
* When a tuple is emitted that is anchored on the tuple that arrived: (task id, bolt name, timestamp)

Here is what I can do with the data from above (assuming one can correlate tuples emitted with incoming tuples, based on the anchor):
* For the aggregation bolt, I can look at the distribution of (emit timestamp - incoming timestamp) and see if it makes sense.
* I can graph the life of one tuple, look at spout/bolt vs timestamp graph, and visually see how much time is being spent in each bolt, as well as how much time is spent in the Storm infrastructure / ZMQ.
This data can be overlayed for multiple tuples to get a sense of the timing distribution for the topology.
* Using the task ID information, one can do a cool overlay graph that traces the distribution of a number of tuples over a topology. One can use that to see if field groupings are working, are unevenly distributed, etc.

For now I may start implementing this idea in scala-storm DSL.