reddmetrics/forma-clj — pull request #240 by dpetrovics (2012-09-07 05:06:53+00:00)
Title: added lein-emr to project.clj and added bootstrap actions config xml file

Hey guys,
So I created a new, totally revamped version of forma-deploy suitable for open source release. I made it as a lein plugin and called it [lein-emr](https://github.com/dpetrovics/lein-emr). Its up on clojars. The library is completely general and doesn't contain any details specific to the FORMA project. We can definitely make it a redd-metrics library if you guys like it :)

To use it, all you have to do (from within the forma-clj directory):

lein emr -t large -s 10 -b 0.15
OR
lein emr --type large --size 10 --bid 0.15

Of course it'll work inside any project that has the plugin or if your user profile has the plugin, not just forma-clj. All of the usual options from forma-deploy are available. You can type 'lein emr help" to see all the options.

The only options which don't make sense to specify via the command line are the bootstrap actions. In forma-deploy they were hardcoded into the boot-emr! function. Now, anybody can just create an xml file in their project directory with up to 16 bootstrap actions and arguments (the various predefined bootstrap actions, written by amazon, are described
[here](http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/Bootstrap.html). You can also put in custom bootstrap actions like @robinkraft 's script on s3. In this pull request I put the config file in the root project directory as bsaconfig.xml -- hopefully pretty self-explanatory.

There were a whole bunch of other options specified in the code that I felt would make more sense in the config file living up in s3. I called it s3://reddconfig/bootstrap-actions/config_new.xml. You guys can compare it the old config.xml file to see what I added. The 'configure-hadoop' bootstrap action automatically loads all the configs from that file, plus any others that you can specify from within bsaconfig.xml. Since @robinkraft really likes to play around with the max number of map tasks and reduce tasks :), I kept the --mappers and --reducers options as command line options for convenience (internally it just passes these as args to the 'configure-hadoop' bootstrap action, along with the config file from s3.

To summarize-- this works exactly like forma-deploy did, but we can boot the cluster from within forma-clj. I think its also much faster, and the code is way cleaner. Since its general and open source other coders can help improve it, and all of our private configs are provided via 1. command line 2. bsaconfig.xml 3. config_new.xml on s3. We'll change 1) frequently, 2) rarely, and 3) basically never.

Let me know what you guys think! I cranked this out fairly quickly so its possible there are some bugs, although I did my best to check it.

-D