A modification to dynamically create a sparkContext with YARN. sparkR.R modified to pass custom Jar file names and EnvironmentEnv to the sparkConf. RRDD.scala was also modified to accept the new inputs to creatSparkContext.

Usage Example:

Note that in this example SparkR was built with spark 1.1.0 and hadoop 2.3.0.
Follow instruction here https://github.com/amplab-extras/SparkR-pkg#package-installation

Make sure to add the spark assembly jar to classpath and YARN. In this case I used spark-assembly-1.1.0-hadoop2.3.0.jar
Also, very important, the hadoop conf folder needs to be added to classpath. In my case it was /etc/hadoop/conf/

### set class paths ###

.jaddClassPath("/usr/lib/spark/lib/spark-assembly-1.1.0-hadoop2.3.0.jar")
.jaddClassPath("/etc/hadoop/conf/")
.jaddClassPath(“other jar to pass into YARN.jar”)


### set the user system variables for hadoop ###
USER="username"
Sys.setenv(user.name=USER)
Sys.setenv(HADOOP_USER_NAME=USER)

### set variables for the Spark context on yarn ###
context="yarn-client"
appName = "SparkR"
sparkHome = Sys.getenv("SPARK_HOME")

sparkJars=c("other jar to pass into YARN.jar")

sparkEnvir=list(spark.executor.memory=“2G")

sparkExecutorEnv=list(SPARK_YARN_APP_JAR=paste0(.libPaths()[1],"/SparkR/sparkr-assembly-0.1.jar"),
                      SPARK_WORKER_INSTANCES="2",
                      SPARK_WORKER_CORES="4",
                      SPARK_WORKER_MEMORY="2G",
                      SPARK_MASTER_MEMORY="2G",
                      SPARK_YARN_APP_NAME="SparkR",
                      SPARK_YARN_DIST_ARCHIVES="/usr/lib/spark/lib/spark-assembly-1.1.0-hadoop2.3.0.jar")



### start the Spark Context ###
sc <- sparkR.init(master=context,sparkJars=sparkJars,sparkEnvir = sparkEnvir,sparkExecutorEnv=sparkExecutorEnv)
Modification to dynamically create a sparkContext with YARN. Added .setExecutorEnv to the sparkConf in createSparkContext within the RRDD object. This modification was made together with sparkR.R
a few stylistic changes
a few stylistic changes. Also change vars to sparkEnvirMap and eevars to ExecutorEnv, to match sparkR.R
forgot to change hm and ee inside the for loops
added the YARN_CONF_DIR to the classpath
Added LD_LIBRARY_PATH to the ExecutorEnv. This is need so that the nodes can find libjvm.so, or if the master has a different LD_LIBRARY_PATH then the nodes. Make sure to export LD_LIBRARY_PATH  that includes the path to libjvm.so in the nodes.

To Run:

USE_YARN=1 SPARK_YARN_VERSION=2.4.0 SPARK_HADOOP_VERSION=2.4.0 ./install-dev.sh
export YARN_CONF_DIR=<yarn_conf_dir>
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/path/ to/ libjvm.so/ on/ nodes/"
MASTER="yarn-client" ./sparkR


If LD_LIBRARY_PATH not properly set you will find the following error in the yarn logs

Error : .onLoad failed in loadNamespace() for 'rJava', details:
  call: dyn.load(file, DLLpath = DLLpath, ...)
  error: unable to load shared object '/usr/lib/Rlibrary/rJava/libs/rJava.so':
  libjvm.so: cannot open shared object file: No such file or directory
Error: package ‘rJava’ could not be loaded
Execution halted
executorEnv to lower-case e
changed executorEnv to sparkExecutorEnvMap,  will do the same in R
changed ExecutorEnv to sparkExecutorEnvMap, to make it consistent with sparkEnvirMap
Update sparkR.R
Added example to SparkR.R
removed ... from example
Update sparkR.R
Merge pull request #90 from oscaroboto/master

Added ExecutorEnv and ability to pass additional JARs to sparkContext (to be used with YARN)