apache/spark — pull request #1242 by rxin (2014-06-27 05:14:49+00:00)
Title: [SPARK-2304] tera sort example program for shuffle benchmarks

This pull request adds an example program for benchmarking Spark shuffle. It dynamically generates a set of 100 byte records according to the tera sort spec, and repartitions the data based on an evenly spaced range partitioner. By design, it does NOT yet perform sorting after the range partitioning yet.

Some of the code copied directly from Hadoop and simplified (the data generator stuff).

I've used this utility to benchmark Spark at scale, including performing 100TB of shuffle in 12 mins on 290 nodes.