Shopify/spark — pull request #1 by honkfestival (2014-03-29 15:45:41+00:00)
Title: Pyspark mem limit

Until we get a better handle on runaway python processes, I'd like to be able to limit the amount of heap space available for a given python worker. This PR gives us the option to pull it from the environment variable `SPARK_PYTHON_MAX_HEAP` so that we can have node-specific configs and otherwise default to 32 gigs.

The default limit is `(-1, -1)` which is also `(9223372036854775807, 9223372036854775807)` which is 9.223EB (exabytes!). The only real difference between the soft and hard limits is that the soft limit can be modified at runtime as long as it stays under the hard limit.

For reference: http://docs.python.org/2/library/resource.html

Limiting pyspark memory has the additional benefit of not crowding the other daemons (HDFS, etc) running on the boxes, which will be problematic until we can get proper isolation via Mesos or whatever.

@Shopify/data-engineers what do you think?