add Python streaming tests
Report Spark Core results as JSON:

After this patch, the "result: " line printed to the stdout logs will contain
a JSON object that captures information on the test configuration in addition
to the actual test results.  This makes it easier to process the test results
using tools like Pandas or Spark SQL.
Extend SchedulerThroughputTest to test closure size, number of jobs.

I’m adding these parameters in order to test the broadcast-based
task shipping added in Spark 1.1.0.
Merge pull request #18 from tdas/master

Updated Streaming Perf tests with Receivers
Make setting configs more user-friendly

Previously there were tons of flags that the user must set, and
these are scattered throughout the config.py file. This commit
organizes the necessary settings by category and moves the
important ones to the top.

It also replaces all "SKIP_*" flags with the positive form,
which makes it more straightforward for the user to reason about.
Always rsync Spark to all nodes

When a Worker config changes, for instance, we need to manually
rsync Spark before running spark-perf. With this commit this is
no longer the case.
Ignore results directory
Actually skip PySpark tests if they are not configured to run
Merge pull request #20 from davies/streaming

add Python streaming tests
Add header to result file to explain what the numbers mean

It was very confusing before. For each test all we see is something
like 10.1, 0.25, 9.75, 10.8, 10.35. Every time I have to dig into
the code to figure out what these mean. This commit adds a header to
the result file to explain this.
Rewrite README; add bin/spark-ec2/install-maven script.
Add support for Yarn

This is almost trivial. The only thing we need is to set
SPARK_CLUSTER_URL to "yarn", and avoid restarting the standalone
cluster as we normally do
Support running on Mesos
Move mesos configuration to the top
Remove unused imports in mesos cluster
Merge pull request #26 from tnachen/mesos_support

Support running on Mesos
Fix config and add tests for Broadcast

1. fix the pyspark tests in config.py.template
2. add a option to disable rsync SPARK_HOME
3. add two tests for python broadcast
4. dump the configs and results as json
5. some refactor
6. add --list and --al

Author: Davies Liu <davies@databricks.com>
Author: Davies Liu <davies.liu@gmail.com>

Closes #28 from davies/json and squashes the following commits:

71ebc9b [Davies Liu] add --list and --all
5773ecb [Davies Liu] cleanup
3641ce1 [Davies Liu] fix config
a244e6c [Davies Liu] Merge branch 'master' of https://github.com/databricks/spark-perf into broadcast
def130a [Davies Liu] add test for broadcast
[mllib] All updates for MLlib for QA for Spark 1.2

Major additions:
* added Python tests
* updated tests to do JSON logging
* updated test settings to be more reasonable for QA
* split Scala tests into directories for Spark 1.0, 1.1, 1.2 (since APIs are incompatible)
 * Note: These could share some code, but it is unclear how much we need to support spark-perf for more than the latest 1 or 2 versions of MLlib.

Items remaining (before next Spark release QA period):
* Change Scala testName to match Python test names: “glm-regression” —> GLMRegressionTest
* Make parameter names match. (num-examples, num-rows, etc.)
* Refactor correlation tests so pearson/spearman is a parameter.
* Better data generation in Python

Author: Joseph K. Bradley <joseph@databricks.com>
Author: Joseph K. Bradley <joseph.kurata.bradley@gmail.com>

Closes #29 from jkbradley/qa-1-2 and squashes the following commits:

fde1fef [Joseph K. Bradley] removed mllib from travis.yml since it requires a snapshot of spark
b09b7cb [Joseph K. Bradley] fixed typo
6278838 [Joseph K. Bradley] removed qa config files
e9770eb [Joseph K. Bradley] code cleanups
5a28969 [Joseph K. Bradley] updated config for mllib
802588e [Joseph K. Bradley] added other python tests: kmeans, naive bayes, pearson, spearman
eea334a [Joseph K. Bradley] updated glm debug config
7b1d4b6 [Joseph K. Bradley] removed boosting from spark-perf because of api change.  updated randomforest to fit api change. updated glmdebug config
478fec9 [Joseph K. Bradley] fixed glm debug config
748c97b [Joseph K. Bradley] Added config file for debugging Python regressions.
5b47f34 [Joseph K. Bradley] updated pyspark tests to cache trainRDD before computing testTime on it, to match Scala code
912d641 [Joseph K. Bradley] removed config for als python since we will use the same settings as for scala
1759ddb [Joseph K. Bradley] fixed python json logging to match scala
6335851 [Joseph K. Bradley] made separate python config files since those tests take much longer
3d2b197 [Joseph K. Bradley] updated so shuffle manager is only specified for mllib
a4735bd [Joseph K. Bradley] fixed als and json logging for spark 1.1 python tests
3247fe1 [Joseph K. Bradley] added shuffle manager option to all config files this time
55630e1 [Joseph K. Bradley] Added json logging for python mllib tests
26f2d49 [Joseph K. Bradley] decreased svd test size config
05524cc [Joseph K. Bradley] json logging now in all of mllib tests.  fixed decision tree test bug for 1.2
95af6cd [Joseph K. Bradley] added json logging for mllib tests for spark 1.1
160f79f [Joseph K. Bradley] added json logging for mllib tests for spark 1.2.  need to do 1.1 and 1.0 still
ad44fac [Joseph K. Bradley] fixed naive bayes config.  fixed loss for mllib_test.py regression test
81d9ae4 [Joseph K. Bradley] fixed config file
0ce6405 [Joseph K. Bradley] updated config scripts for mllib
7bf7210 [Joseph K. Bradley] fixed python mllib als test
cd3f0d2 [Joseph K. Bradley] added python mllib als test
c8d016a [Joseph K. Bradley] added pyspark mllib glm regression test
fb37915 [Joseph K. Bradley] fixed pyspark mllib glm classification test
d9d1ce9 [Joseph K. Bradley] added initial pyspark mllib test but not yet tested
b900032 [Joseph K. Bradley] changed als test to set blocks to num rdd partitions
517b861 [Joseph K. Bradley] updated config files
dda5c17 [Joseph K. Bradley] added base config for qa for 1.1
645eaf6 [Joseph K. Bradley] updated config.py.template
1da771f [Joseph K. Bradley] Major refactoring of MLlib part of spark-perf. * Added structure to GLM part of tests, eliminating some code * Added 3 separate directories with complete tests for Spark 1.0, 1.1, 1.2 * Added GradientBoosting test as part of DecisionTreeTests
7ecd2c5 [Joseph K. Bradley] Updated DecisionTree test config options to a reasonable QA set. Added SNAPSHOT suffix to MLlibTestsBuild.scala for onepointone.  Fixed mllib jar name in testsuites.py.  Updated to use WeightedEnsembleModel instead of RandomForestModel.  Updated cluster.py slave file check.
04ff018 [Joseph K. Bradley] fixed off-by-one bug in computing median
841052c [Joseph K. Bradley] Fixed bug in loading data for DecisionTree tests (training data was loaded as test data)
00bdbcf [Joseph K. Bradley] Added DecisionTree test options to support testing on a real dataset: * training-data, test-data: training, test dataset paths or URIs Also added DecisionTree test option: * test-data-fraction: fraction of data to hold out for training (used for both real & synthetic data)
c39b32a [Joseph K. Bradley] prep to merge with master
c53c685 [Joseph K. Bradley] removed streaming tests, added DT one
e6553f3 [Joseph K. Bradley] Updated DecisionTree test to take featureSubsetStrategy parameter.
6ed8ad2 [Joseph K. Bradley] Updated to run RandomForest with numTrees parameter
Merge remote-tracking branch 'databricks/master' into driver-ha

Conflicts:
	lib/sparkperf/main.py