mesos/spark — Issue #105: Using 0 partition RDD in a shuffle causes crashes
Author: woggling
Created: 2011-12-25 20:20:28+00:00
State: open

If one uses a zero partition RDD (e.g. the result of creating a Hadoop RDD from empty files with certain input formats) as part of a shuffle dependency, spark will get confused: DAGScheduler realizes that it doesn't need to realize the RDD, but it won't tell MapOutputTracker about it, so MapOutputTracker will return null when the reduce tasks ask for the shuffle locations.