SparkHadoopWriter allows the user to specify the jobId, though here was set it to be stageId. What are the requirements on JobID? It is possible that multiple HadoopRDDs actually output the same JobId, for instance, if they're connected via a narrow dependency (and thus within the same stage). Is that a problem?