reddmetrics/forma-clj — pull request #328 by robinkraft (2012-12-25 07:14:28+00:00)
Title: workflow automation

This pull request contains a working bash script for testing the full workflow with real, raw data for tile `[28 8]`. For each step there are new defmains and full tests for all steps after preprocessing, as well as one big test of the full workflow using dummy data.

The `./dev/run-forma.sh` script is hardcoded to point at selected data already on S3, so you can run the entire workflow from raw rain, MODIS and fires data through probability series in about 3 hours using a single high-memory instance. It wouldn't work on all the data since certain steps need different cluster configurations, but it's close. And the defmains it uses will be the building blocks for regular updates of the FORMA data.

There is a lot of room for refinement in how we launch the processing and how the defmains are designed (they're all pretty similar), so now that it all works (see output image below) I want to get some eyes on it ASAP.

Happy holidays!

![Google ChromeScreenSnapz001](https://f.cloud.github.com/assets/428784/30309/7c4aef46-4e62-11e2-974b-64851d4df99b.png)