apache/spark — pull request #377 by patmcdonough (2014-04-10 06:30:09+00:00)
Title: [WIP] Spark-1392: Add parameter to reserve minimum memory for the system and increase default executor memory

* create a new property that sets the minimum amount of heap reserved for the system/application: _spark.system.memoryReservedSize_
* account for that value prior to calculating memory available for storage and shuffle
* set the new property at 300m by default (based on what we are seeing in a local spark-shell running JDK7 with the spark-0.9.0-hadoop-2 binary distribution)
* increase the default spark.executor.memory beyond 512m since we are going to reserve over half of that for spark itself

This is marked as WIP as testing is still in progress. I was still able to cause an OOM by running a count distinct on the dataset linked to in the JIRA, but bumping _spark.system.memoryReservedSize_ to 350m prevented it.