apache/spark — pull request #257 by NathanHowell (2014-03-27 15:13:30+00:00)
Title: Minimize the scope of runAsUser to support Hadoop token passing

To support accessing a secure HDFS installation from Mesos executed Spark jobs, the resources attached to the `SparkContext` should be fetched before invoking `UserGroupInformation.doAs`. Or at least the credentials should be fetched before `doAs`. This allows us to set the `HADOOP_TOKEN_FILE_LOCATION` environment variable on the executor to a token cache file generated for each job.

`UserGroupInformation.doAs` fails to execute if the environment variable is set but the file does not exist, and trying to stuff the credentials back in later via some other mechanism is problematic. This approach is similar to how credentials are passed for YARN executed jobs.