Just for the record, 100k partitions coalesced into 2k takes 850ms on my MBA. Doing it in bulk didn't buy much, perhaps 10ms on average (repeating the experiment 20 times to get the std.dev down to negligible numbers). So for now, we're making one call to DAGScheduler for each partition of the RDD.