Merge pull request #85 from edwardt/master

Make spark verison configurable in build script per ISSUE122
[SPARKR-125] Get the iterator of the parent RDD before launching a R worker process in compute() of RRDD/PairwiseRRDD
Adds function distinct() and mapValues().
typo fixed.
Fixes style and adds an optional param 'numPartition' in distinct().
[SPARKR-108] Implement map-side reduction for reduceByKey().
Merge pull request #88 from hlin09/hlin09

Adds two functions from Spark: distinct() and mapValues().
Merge pull request #89 from sun-rui/SPARKR-108

[SPARKR-108] Implement map-side reduction for reduceByKey().
Merge pull request #87 from sun-rui/SPARKR-125

[SPARKR-125] Get the iterator of the parent RDD before launching a R wor...
A modification to dynamically create a sparkContext with YARN. sparkR.R modified to pass custom Jar file names and EnvironmentEnv to the sparkConf. RRDD.scala was also modified to accept the new inputs to creatSparkContext.

Usage Example:

Note that in this example SparkR was built with spark 1.1.0 and hadoop 2.3.0.
Follow instruction here https://github.com/amplab-extras/SparkR-pkg#package-installation

Make sure to add the spark assembly jar to classpath and YARN. In this case I used spark-assembly-1.1.0-hadoop2.3.0.jar
Also, very important, the hadoop conf folder needs to be added to classpath. In my case it was /etc/hadoop/conf/

### set class paths ###

.jaddClassPath("/usr/lib/spark/lib/spark-assembly-1.1.0-hadoop2.3.0.jar")
.jaddClassPath("/etc/hadoop/conf/")
.jaddClassPath(“other jar to pass into YARN.jar”)


### set the user system variables for hadoop ###
USER="username"
Sys.setenv(user.name=USER)
Sys.setenv(HADOOP_USER_NAME=USER)

### set variables for the Spark context on yarn ###
context="yarn-client"
appName = "SparkR"
sparkHome = Sys.getenv("SPARK_HOME")

sparkJars=c("other jar to pass into YARN.jar")

sparkEnvir=list(spark.executor.memory=“2G")

sparkExecutorEnv=list(SPARK_YARN_APP_JAR=paste0(.libPaths()[1],"/SparkR/sparkr-assembly-0.1.jar"),
                      SPARK_WORKER_INSTANCES="2",
                      SPARK_WORKER_CORES="4",
                      SPARK_WORKER_MEMORY="2G",
                      SPARK_MASTER_MEMORY="2G",
                      SPARK_YARN_APP_NAME="SparkR",
                      SPARK_YARN_DIST_ARCHIVES="/usr/lib/spark/lib/spark-assembly-1.1.0-hadoop2.3.0.jar")



### start the Spark Context ###
sc <- sparkR.init(master=context,sparkJars=sparkJars,sparkEnvir = sparkEnvir,sparkExecutorEnv=sparkExecutorEnv)
Modification to dynamically create a sparkContext with YARN. Added .setExecutorEnv to the sparkConf in createSparkContext within the RRDD object. This modification was made together with sparkR.R
a few stylistic changes
a few stylistic changes. Also change vars to sparkEnvirMap and eevars to ExecutorEnv, to match sparkR.R
Changes to make SparkR work with YARN
Add maximum() and minimum() API to RDD.
Fix SBT build, add dependency tree plugin
Add dependency on spark yarn module
Don't use quotes around yarn
Consolidate yarn flags
use reduce() to implemement max() and min().