apache/spark — pull request #3003 by davies (2014-10-29 21:10:33+00:00)
Title: [SPARK-3466] Limit size of results that a driver collects for each action

Right now, operations like collect() and take() can crash the driver with an OOM if they bring back too many data.

This PR will introduce spark.driver.maxResultSize, after setting it, the driver will abort a job if its result is bigger than it.

By default, it's unlimited, for backward compatibility.

In local mode, the driver and executor share the same JVM, and the default JVM heap size is too small, we can not set a proper limit for it as default, or it can not protect JVM from OOM, or it's too small for normal usage.