A modification to dynamically create a sparkContext with YARN. sparkR.R modified to pass custom Jar file names and EnvironmentEnv to the sparkConf. RRDD.scala was also modified to accept the new inputs to creatSparkContext.

Usage Example:

Note that in this example SparkR was built with spark 1.1.0 and hadoop 2.3.0.
Follow instruction here https://github.com/amplab-extras/SparkR-pkg#package-installation

Make sure to add the spark assembly jar to classpath and YARN. In this case I used spark-assembly-1.1.0-hadoop2.3.0.jar
Also, very important, the hadoop conf folder needs to be added to classpath. In my case it was /etc/hadoop/conf/

### set class paths ###

.jaddClassPath("/usr/lib/spark/lib/spark-assembly-1.1.0-hadoop2.3.0.jar")
.jaddClassPath("/etc/hadoop/conf/")
.jaddClassPath(“other jar to pass into YARN.jar”)


### set the user system variables for hadoop ###
USER="username"
Sys.setenv(user.name=USER)
Sys.setenv(HADOOP_USER_NAME=USER)

### set variables for the Spark context on yarn ###
context="yarn-client"
appName = "SparkR"
sparkHome = Sys.getenv("SPARK_HOME")

sparkJars=c("other jar to pass into YARN.jar")

sparkEnvir=list(spark.executor.memory=“2G")

sparkExecutorEnv=list(SPARK_YARN_APP_JAR=paste0(.libPaths()[1],"/SparkR/sparkr-assembly-0.1.jar"),
                      SPARK_WORKER_INSTANCES="2",
                      SPARK_WORKER_CORES="4",
                      SPARK_WORKER_MEMORY="2G",
                      SPARK_MASTER_MEMORY="2G",
                      SPARK_YARN_APP_NAME="SparkR",
                      SPARK_YARN_DIST_ARCHIVES="/usr/lib/spark/lib/spark-assembly-1.1.0-hadoop2.3.0.jar")



### start the Spark Context ###
sc <- sparkR.init(master=context,sparkJars=sparkJars,sparkEnvir = sparkEnvir,sparkExecutorEnv=sparkExecutorEnv)
Modification to dynamically create a sparkContext with YARN. Added .setExecutorEnv to the sparkConf in createSparkContext within the RRDD object. This modification was made together with sparkR.R
a few stylistic changes
a few stylistic changes. Also change vars to sparkEnvirMap and eevars to ExecutorEnv, to match sparkR.R
Changes to make SparkR work with YARN
Fix SBT build, add dependency tree plugin
Add dependency on spark yarn module
Don't use quotes around yarn
Consolidate yarn flags
forgot to change hm and ee inside the for loops
added the YARN_CONF_DIR to the classpath
Added LD_LIBRARY_PATH to the ExecutorEnv. This is need so that the nodes can find libjvm.so, or if the master has a different LD_LIBRARY_PATH then the nodes. Make sure to export LD_LIBRARY_PATH  that includes the path to libjvm.so in the nodes.

To Run:

USE_YARN=1 SPARK_YARN_VERSION=2.4.0 SPARK_HADOOP_VERSION=2.4.0 ./install-dev.sh
export YARN_CONF_DIR=<yarn_conf_dir>
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/path/ to/ libjvm.so/ on/ nodes/"
MASTER="yarn-client" ./sparkR


If LD_LIBRARY_PATH not properly set you will find the following error in the yarn logs

Error : .onLoad failed in loadNamespace() for 'rJava', details:
  call: dyn.load(file, DLLpath = DLLpath, ...)
  error: unable to load shared object '/usr/lib/Rlibrary/rJava/libs/rJava.so':
  libjvm.so: cannot open shared object file: No such file or directory
Error: package ‘rJava’ could not be loaded
Execution halted
executorEnv to lower-case e
changed executorEnv to sparkExecutorEnvMap,  will do the same in R
changed ExecutorEnv to sparkExecutorEnvMap, to make it consistent with sparkEnvirMap
Remove SBT plugin
Update sparkR.R
Added example to SparkR.R
removed ... from example
Use temporary files for input instead of stdin
This fixes a bug for Windows where stdin would get truncated