Initial build and project layout
ignore intellij files
Rough cut of a test based on data similar to the benchmark.
Various improvements and cleanup
Adding unit test for data generation
Adding test runner and some other changes
Removing explicit Spark dep
Port Denny's spark-perf (which is in ruby) to python!
Updates to python test framework - runs but output needs cleanup.
Use stdout for framework messages (instead of stderr) and print results in csv format
to a filename which is now a required arg when running bin/run. Add parsing and logic
to bin/run that captures output from scala TestRunner and computes metrics based on it
(median and variance) and prints those as CSV.
Uncomment the line in bin/run that does git reset to hash specified at command line.
I had commented this out before to shorten dev cycle.
Add README file with instructions for running, add spark-ec2/copy-dir
which is a copy from the same file in the https://github.com/mesos/spark-ec2 repo,
and update bin/run to use this copy-dir (Denny's had used mesos-ec2).
Include sbt instead of requiring user to have installed it on their own.
Minor updates to README, remove jvm options from sbt_cmd in bin/run
since we should just consolidate those in sbt/sbt script (where some
were already specified) and added a print statement at end of tests
saying all done.
Actually use SPARK_MASTER
Better handling of failed tests
Removing un-used environment variables
STASH - First cut at scheduler
Factoring out command running logic
mesos-ec2 --> spark-ec2