apache/incubator-spark — pull request #615 by aarondav (2014-02-18 20:09:11+00:00)
Title: SPARK-929: Fully deprecate usage of SPARK_MEM

This patch cements our deprecation of the SPARK_MEM environment variable by replacing it with three more specialized variables:
SPARK_DAEMON_MEMORY, SPARK_EXECUTOR_MEM, and SPARK_DRIVER_MEM

The creation of the latter two variables means that we can safely set driver/job memory without accidentally setting the executor memory.
SPARK_EXECUTOR_MEM is not actually public -- it is only used by the Mesos scheduler (and set within SparkContext). The proper way of configuring executor memory is through the "spark.executor.memory" property.

SPARK_DRIVER_MEM is a new public variable, which is needed because there is currently no public way of setting the memory of jobs launched by spark-class.

Other memory considerations:
- The repl's memory can be set through the "--drivermem" command-line option, which really just sets SPARK_DRIVER_MEM.
- run-example doesn't use spark-class, so the only way to modify examples' memory is actually an unusual use of SPARK_JAVA_OPTS (which is normally overriden in all cases by spark-class).

This patch also fixes a lurking bug where spark-shell misused spark-class (the first argument is supposed to be the main class name, not java options), as well as a bug in the Windows spark-class2.cmd. I have not yet tested this patch on either Windows or Mesos, however.