apache/incubator-spark — pull request #626 by CodingCat (2014-02-21 05:44:05+00:00)
Title: [SPARK-1100] prevent Spark from overwriting directory silently and leaving dirty directory

Thanks for Diana Carroll to report this issue

the current saveAsTextFile/SequenceFile will overwrite the output directory silently if the directory already exists, this behaviour is not desirable because

1. overwriting the data silently is not user-friendly

2. if the partition number of two writing operation changed, then the output directory will contain the results generated by two runnings

My fix includes:

1. add some new APIs with a flag for users to define whether he/she wants to overwrite the directory:

if the flag is set to true, then the output directory is deleted first and then written into the new data to prevent the output directory contains results from multiple rounds of running;

if the flag is set to false, Spark will throw an exception if the output directory already exists

2. I didn't change saveNewHadoopAPI because in the new API, the overwrite flag is defined by the implementation of RecordWriter, we don't need to control that in Spark

3. changed JavaAPI part

4. default behaviour is overwriting

-----

Two questions

1. should we deprecate the old APIs without such a flag?

2. I noticed that Spark Streaming also called these APIs, I thought we don't need to change the related part in streaming? @tdas