jtriley/StarCluster — pull request #236 by adgaudio (2013-03-27 18:44:20+00:00)
Title: AWS VPC support

This PR revives work by @jtriley and @locojay to use StarCluster inside Amazon VPC and adds a couple bug fixes. I have been using this successfully for about a week with no significant problems.

Changes and details:

- Security groups start with `sc-` rather than `@sc-`, which means that existing cluster groups will be unrecognizable by starcluster, making this PR **backwards incompatible**.
- Due to some problems with Amazon's api and removing security groups, VPC clusters that are immediately terminated will not properly delete their security groups. The appropriate approach here is to `stop` a cluster and then `terminate` it a few minutes later, or to `terminate` and then `terminate -f` a few minutes later.
- When running in a vpc, the public "dns_name" is by default unavailable, which may make some StarCluster features fail when using the VPC.
- If using VPC, the clusters must be started (and ssh'd into) from inside the vpc. Currently, we do not support ssh tunneling from outside the VPC (and we probably shouldn't) .
- If using VPC clusters with other nodes in the vpc, users may need to configure their security groups using this plugin: https://gist.github.com/adgaudio/5208195

I'd love to get input on next steps with this.