databricks/spark-perf — pull request #2 by davies (2014-08-07 05:39:25+00:00)
Title: add performance tests for pyspark

The test cases are copied from spark tests

Here is local tests results (scale=0.01):

python-scheduling-throughput, SchedulerThroughputTest --num-tasks=5000 --num-trials=4 --inter-trial-wait=1, 17.284, 0.627, 16.029, 17.284, 16.029
python-agg-by-key, AggregateByKey --num-trials=4 --inter-trial-wait=1 --num-partitions=4 --reduce-tasks=4 --random-seed=5 --persistent-type=memory --num-records=2000000 --unique-keys=200 --key-length=10 --unique-values=10000 --value-length=10 --storage-location=/spark-perf-kv-data, 1.662, 0.011, 1.64, 1.640, 1.662
python-agg-by-key-int, AggregateByKeyInt --num-trials=4 --inter-trial-wait=1 --num-partitions=8 --reduce-tasks=8 --random-seed=5 --persistent-type=memory --num-records=4000000 --unique-keys=400 --key-length=10 --unique-values=20000 --value-length=10 --storage-location=/spark-perf-kv-data, 1.428, 0.003, 1.421, 1.421, 1.428
python-agg-by-key-naive, AggregateByKeyNaive --num-trials=4 --inter-trial-wait=1 --num-partitions=4 --reduce-tasks=4 --random-seed=5 --persistent-type=memory --num-records=2000000 --unique-keys=200 --key-length=10 --unique-values=10000 --value-length=10 --storage-location=/spark-perf-kv-data, 1.917, 0.008, 1.901, 1.917, 1.901
python-sort-by-key, SortByKey --num-trials=4 --inter-trial-wait=1 --num-partitions=1 --reduce-tasks=1 --random-seed=5 --persistent-type=memory --num-records=200000 --unique-keys=20 --key-length=10 --unique-values=1000 --value-length=10 --storage-location=/spark-perf-kv-data, 0.751, 0.018, 0.716, 0.751, 0.716
python-sort-by-key-int, SortByKeyInt --num-trials=4 --inter-trial-wait=1 --num-partitions=1 --reduce-tasks=1 --random-seed=5 --persistent-type=memory --num-records=400000 --unique-keys=40 --key-length=10 --unique-values=2000 --value-length=10 --storage-location=/spark-perf-kv-data, 1.295, 0.020, 1.254, 1.254, 1.295
python-count, Count --num-trials=4 --inter-trial-wait=1 --num-partitions=4 --reduce-tasks=4 --random-seed=5 --persistent-type=memory --num-records=2000000 --unique-keys=200 --key-length=10 --unique-values=10000 --value-length=10 --storage-location=/spark-perf-kv-data, 0.337, 0.002, 0.334, 0.334, 0.337
python-count-w-fltr, CountWithFilter --num-trials=4 --inter-trial-wait=1 --num-partitions=4 --reduce-tasks=4 --random-seed=5 --persistent-type=memory --num-records=2000000 --unique-keys=200 --key-length=10 --unique-values=10000 --value-length=10 --storage-location=/spark-perf-kv-data, 1.144, 0.013, 1.118, 1.118, 1.144