mesos/spark — pull request #303 by nkronenfeld (2012-11-12 19:24:01+00:00)
Title: Add capabilities to RDD's to show the DAG structure for debugging

This is my attempt to output the DAG structure of an RDD, as per the "RDD Task Introspection" thread I started on spark-users.

You use it by just adding a .debug() call to any RDD in the chain about to be calculated, i.e.,

val foo = sc.textFile(...).map(...).debug().collect()

The DAGScheduler checks the stage, before it runs it, to see if any of its tasks are marked debug, and if they are, it prints out their structure.

I've only run this in a local setup, but if I understand things correctly, the scheduler is only run locally anyway, so I don't think there should be any issues there.

This also introduces a level in the RDD class tree in which to consolidate previous-RDD methods, which I suspect will end up being needed anyway eventually; I'm not sure if I've done this the best way or not, and welcome comments.