apache/spark — pull request #1994 by pwendell (2014-08-17 03:05:55+00:00)
Title: SPARK-2881: Avoid collisisions in Snappy staging directory.

By default Snappy uses java.io.tempdir for copying the Snappy native library.
If two users run Spark jobs on the same machine it can cause an exception when
the second user tries to access or overwrite the snappy file created by the
first user. This will fail Spark jobs out-of-the-box if they are run on a
machine shared by different users.

Snappy does expose a mechanism to customize the temp directory via a system
property. This system property is read in a static block inside of Snappy code.

Snappy-java fixes this in newer versions. We can upgrade snappy-java in master
but this proposes a "best effort" fix for 1.1 where we try to set the system
property in a static block before Snappy reads it. I've tested it and it does
work, but it relies on static initialization order which is brittle. I.e.
if user code accesses Snappy libraries first this could not-work.

An alternative work-around for users is to explicitly set

org.xerial.snappy.tempdir

themselves through Spark's java options. I also filed a bug upstream with
Snappy-java to ask them for better behavior here:

https://github.com/xerial/snappy-java/issues/84

I think this is worth merging because in many cases it will fix the issue and
at worst it's a no-op.