Leolh/spark — pull request #1 by Leolh (2014-08-26 07:06:17+00:00)
Title: [SPARK-3228][Streaming]

When I use DStream to save files to hdfs, it will create a directory and a empty file named "_SUCCESS" for each job which made in the batch duration.
But if there are no data from source for a long time , and the duration is very short(e.g. 10s), it will create so many directory and empty files in hdfs.
I don't think it is necessary. So I want to modify class DStream's method saveAsObjectFiles and saveAsTextFiles , it creates directory and files just when the RDD's partitions size > 0 .