apache/incubator-spark — pull request #456 by liyinan926 (2014-01-18 01:34:52+00:00)
Title: Allow files added through SparkContext.addFile() to be overwritten

This is useful for the cases when a file needs to be refreshed and downloaded by the executors periodically. For example, a possible use case is: the driver periodically refreshes a Hadoop delegation token and writes it to a token file. The token file needs to be downloaded by the executors whenever it gets refreshed. However, the current implementation throws an exception when the target file exists and its contents do not match those of the new source. This PR adds an option to allow files to be overwritten to support use cases similar to the above.