apache/spark — pull request #1595 by falaki (2014-07-25 23:06:22+00:00)
Title: [Core][SPARK-2696] Reduce default value of spark.serializer.objectStreamReset

The current default value of spark.serializer.objectStreamReset is 10,000.
When trying to re-partition (e.g., to 64 partitions) a large file (e.g., 500MB), containing 1MB records, the serializer will cache 10000 x 1MB x 64 ~= 640 GB which will cause out of memory errors.

This patch sets the default value to a more reasonable default value (100).