infochimps/cluster_chef — Issue #75: cluster_chef tools launch vagrant instances
Author: mrflip
Created: 2011-11-30 17:19:13+00:00
State: open
Labels: version_3_cleanup

Cluster chef should launch vagrants just as well as ec2 instances:

1. Create/Update `Vagrantfile`s into persistent (versionable) directory tree
2. `Vagrantfile`s should come from a (configurable) `.erb` template
3. Goal should be that any reasonable cluster file works with no changes (apart from hints in its `cloud` block) on vagrant vs. ec2.

#### Notes:

* this should pretty well clarify #66 (`cloud` keyword FTL) and show the way towards #28 (cloud agnostic)
* Initial effort should assume a chef server exists, as the discovery features etc are pretty lame without them. We can get less opinionate later.

#### Same/Alternate Universes

The only near-term internal need for VMs at infochimps is testing/development: we want to be able to define a "foobar" cluster, run it a bunch of times as VMs (ensuring that it does NOT interact with the production machines, which live in the EC2 cloud), then throw a switch and deploy it to the cloud. VM launches should thus be isolated but behave as closely as possible to their eventual cloud selves.

So a person might argue that we should have one true chef server, do something like prefix cluster names with `vm_` to mark them as second-class citizens, and in other ways use code + convention to hold them semi-isolated. However:
* when I look at the chef server, I'll see nodes that I have no access to.
* If two people are stunting the same cluster on their own machines they'll collide on the chef server
* I predict this will lead to conditional code in all sorts of wrong places
* there would be no way to use production VMs.
* the phrase "semi-isolated" reads to me as "yeah, until you find out it isn't".

I think the right plan is the One Incestuous Family plan:
* VMs in same chef server are all held equal.
* Clusters have a preferred cloud, but launching some eg. cassandra nodes as VMs and some as cloud nodes is totally fine -- they'll try to discover each other, and once discovered will start pulling in shards. This accomodates people using VMs in production environments, and terrifies people using VMs for testing purposes.
* For our use case VMs and cloud machines are held truly isolated by giving each their own chef server. To discover cloud state (eg have them hook up to a staging DB), we mirror those nodes' metadata to the stunt VM chef server.
* We let you munge the chef Environment name with the cloud name. Say EC2 is my primary cloud, and my cluster file declares `environment(:prod)`. When I launch an EC2 server, it is placed into the `prod` Environment; when I launch a VM it is placed into the `vm_prod` environment. This gives you some ability to isolate/tailor VM instances in a shared universe. For our purposes, this would let us cripple any VM that somehow found its way to the production chef server.