Rewrote most of the code, added spot instance manager, all tasks are now defined in the fab file. removed GAE code, it is now a simple marketing page.
rewrote spot manager. moved job setup to fab file.
Converted GAE into a generic server, added task for launching spot instance
tested on a spot instance
Added valid until to spot request, used write_batch instead of write in queue
Added new crawls, tested code, completed execution on the first 2013 crawl.
added some documentation and example.json