A modification to dynamically create a sparkContext with YARN. sparkR.R modified to pass custom Jar file names and EnvironmentEnv to the sparkConf. RRDD.scala was also modified to accept the new inputs to creatSparkContext.

Usage Example:

Note that in this example SparkR was built with spark 1.1.0 and hadoop 2.3.0.
Follow instruction here https://github.com/amplab-extras/SparkR-pkg#package-installation

Make sure to add the spark assembly jar to classpath and YARN. In this case I used spark-assembly-1.1.0-hadoop2.3.0.jar
Also, very important, the hadoop conf folder needs to be added to classpath. In my case it was /etc/hadoop/conf/

### set class paths ###

.jaddClassPath("/usr/lib/spark/lib/spark-assembly-1.1.0-hadoop2.3.0.jar")
.jaddClassPath("/etc/hadoop/conf/")
.jaddClassPath(“other jar to pass into YARN.jar”)


### set the user system variables for hadoop ###
USER="username"
Sys.setenv(user.name=USER)
Sys.setenv(HADOOP_USER_NAME=USER)

### set variables for the Spark context on yarn ###
context="yarn-client"
appName = "SparkR"
sparkHome = Sys.getenv("SPARK_HOME")

sparkJars=c("other jar to pass into YARN.jar")

sparkEnvir=list(spark.executor.memory=“2G")

sparkExecutorEnv=list(SPARK_YARN_APP_JAR=paste0(.libPaths()[1],"/SparkR/sparkr-assembly-0.1.jar"),
                      SPARK_WORKER_INSTANCES="2",
                      SPARK_WORKER_CORES="4",
                      SPARK_WORKER_MEMORY="2G",
                      SPARK_MASTER_MEMORY="2G",
                      SPARK_YARN_APP_NAME="SparkR",
                      SPARK_YARN_DIST_ARCHIVES="/usr/lib/spark/lib/spark-assembly-1.1.0-hadoop2.3.0.jar")



### start the Spark Context ###
sc <- sparkR.init(master=context,sparkJars=sparkJars,sparkEnvir = sparkEnvir,sparkExecutorEnv=sparkExecutorEnv)