mesos/spark — pull request #345 by JoshRosen (2012-12-29 01:15:42+00:00)
Title: Fix deletion of files in current working directory by clearFiles()

This addresses an issue where Spark could delete files in the current working directory that were added to the job using `addFile()`. I encountered this issue when working on PySpark's code deployment mechanism, which is based on `addFile()`.

From user-code's perspective, files added through `addFile()` are assumed to be in the current working directory. For jobs that are run locally using `DAGScheduler.runLocally()`, tasks run with the driver's current working directory. As a result, files added through `addFile()` must be copied to the driver's current working directory. There's no mechanism to change the CWD in Java.

`clearFiles()` and `clearJars()` clean up these files when the driver exits. This can be a problem if the original files that were added were in the driver's current working directory, because this will cause them to be deleted.

A long-term fix would be to hide the location of fetched files from user code by requiring it to access files through an API like `SparkFiles.get("my-file-name.txt")`. This will require changes to user code and may require changes to Shark.

As a short-term fix, this pull request removes the code that deletes files in the current working directory and adds checks to `Utils.fetchFiles()` to avoid overwriting existing local files with new data. The one downside of this change is that it may add junk to the current working directory. This is preferable to accidentally deleting files.

I've also added `addFile()`/`addJar()` to the Java API.

I also added synchronization to `LocalScheduler.updateDependencies` to avoid performing multiple parallel fetches for the same file.