apache/incubator-spark — pull request #311 by tmyklebu (2013-12-29 04:54:54+00:00)
Title: SPARK-991:  Report information gleaned from a Python stacktrace in the UI

Scala:
- Added setCallSite/clearCallSite to SparkContext and JavaSparkContext. These functions mutate a LocalProperty called "externalCallSite."
- Add a wrapper, getCallSite, that checks for an externalCallSite and, if none is found, calls the usual Utils.formatSparkCallSite.
- Change everything that calls Utils.formatSparkCallSite to call getCallSite instead. Except getCallSite.
- Add wrappers to setCallSite/clearCallSite wrappers to JavaSparkContext.

Python:
- Add a gruesome hack to rdd.py that inspects the traceback and guesses what you want to see in the UI.
- Add a RAII wrapper around said gruesome hack that calls setCallSite/clearCallSite as appropriate.
- Wire said RAII wrapper up around three calls into the Scala code.

I'm not sure that I hit all the spots with the RAII wrapper. I'm also not sure that my gruesome hack does exactly what we want.

One could also approach this change by refactoring runJob/submitJob/runApproximateJob to take a call site, then threading that parameter through everything that needs to know it.

One might object to the pointless-looking wrappers in JavaSparkContext. Unfortunately, I can't directly access the SparkContext from Python---or, if I can, I don't know how---so I need to wrap everything that matters in JavaSparkContext.