amplab/spark-perf — pull request #4 by harveyfeng (2013-09-10 17:40:39+00:00)
Title: Support for building Hive and Shark.

Notes:
* shark and hive repos are cloned to perf-tests/shark and perf-tests/hive. The warehouse directory that Hive uses will be in created in perf-tests/test-warehouses.
* shark.perf.TestRunner can run, but doesn't include functionality yet.

Possible cleanups:
* A ton of 'SPARK' and 'SHARK' prefixes/modifiers have been added to variables in 'config.py', so the script might have lost a bit of readability...same applies to the duplicate code added to the 'run' script. I didn't do major refactorings to address that in this pull request.

* It might be cleaner to separate the configuration file by project - e.g. have a 'spark_config.py' (an almost direct copy of config.py) and 'shark_config.py'. There will be a few duplicate comments if you compare the two files, but I think this is okay, since spark-env.sh and shark-env.sh have a similar formats as well.
The data generation variables (e.g KEY_VAL_TEST_OPTS) could be moved to separate config files as well...