mesos/spark — pull request #603 by pwendell (2013-05-09 06:04:09+00:00)
Title: Several Improvements to EC2 Scripts

This pull request is related to an equivalent request in the shivaram/spark-ec2 repository (https://github.com/shivaram/spark-ec2/pull/1). It should be committed atomically with that. Also - some of the changes are on the AMI image itself.

Together with that change and the new AMI, several improvements have been made:

* It is now possible for users to launch both PVM (standard) and HVM (high capacity) instance architectures. The type will be inferred based on the specific instance type given
* It is now possible for users to launch in any EC2 region
* There is a developer script to auto-deploy new AMI's to all regions
* There is better support for versioning EC2 images as we go forward
* AMI id's are now tracked inside of github
* The image itself has been improved
* Hadoop HDFS and MapReduce upgraded to CDH4
* Hadoop native libraries setup correctly (both for HVM and PVM) along with Snappy compresison
* Matei's public key is no longer present on the image
* Java version 1.7 is added to the machines and made the default

Note that I did not update the Spark/Shark version on the image. They are still at 0.7.0/0.2.0 so this should be a "drop in" replacement for the current AMI.