freeman-lab/thunder — pull request #3 by JoshRosen (2013-12-05 23:58:38+00:00)
Title: Refactor regression package and add unit tests and Travis CI configuration

This pull requests refactors the regression package and adds unit tests. For now, the tests just verify that the code doesn't crash. To allow the regression code to be automatically tested, I refactored the actual job into a function that accepts a SparkContext and arguments, then added a `__main__` method that calls it.

I used [argparse](http://docs.python.org/dev/library/argparse.html) to process command line arguments; it's included in the standard library in Python 2.7+ and available via `pip/easy_install argparse` for earlier Python versions. Down the line, it might be cool to implement some common optional arguments across all of the scripts; for example, there could be a `--serializer` option for using a custom PySpark serializer. Argparse has a lot of other cool features; its [sub-commands](http://docs.python.org/dev/library/argparse.html#sub-commands) feature could be used to implement a single `thunder` script with multiple sub-commands (like `thunder regress`, `thunder tuning`, etc).

To run the tests locally:

```
cd python/test
export SPARK_HOME=...
./run-tests.sh
```

This uses the [nosetests](http://nose.readthedocs.org/en/latest/) test runner.

To convince myself that my refactorings didn't alter the code's behavior, I added a `matdiff.py` script for comparing the .mat files in output directories.

I also added a configuration for automatically running the unit tests on Travis CI. Here's the Travis page for my fork, showing test results from my latest commits: https://travis-ci.org/JoshRosen/thunder

If you want to run the Travis tests on your repository after merging this, log into https://travis-ci.org/ with your GitHub account and follow the instructions to enable the GitHub service integration hook for Travis.