infochimps/cluster_chef — Issue #61: Relocate cookbooks, roles and clusters into separate repo(s)
Author: mrflip
Created: 2011-10-24 02:57:33+00:00
State: open

Cluster chef currently consists of the following separate(able) concerns:

* **cluster_chef tools**
- the DSL that lets you define clusters
- the knife commands which use that DSL
- optional bootstrap scripts for a machine image that can then launch bootstrap-less
* **cluster-oriented cookbooks**
- `cluster_service_discovery` (recipes to let clusters discover services using chef-server's search)
- ?others?
* **cloud utility cookbooks**
- motd, system_internals (swappiness, ulimit, etc)
* **big data cookbooks** (hadoop, cassandra, redis, etc):
- cookbooks
- roles
- clusters

I think it's time to separate those into at least two repos.

REQUEST FOR COMMENTS:

#### Division of concerns

It's clear that the cluster_chef tools and the big data cookbooks should be divorced.

Proposed:

* `cluster_chef` holds only the DSL, knife commands, and bootstrap scripts -- basically the stuff in `lib/`, along with the gemspec etc.
* `cluster_chef-systems` -- holds cookbooks, roles and example clusters that use them.
- Utility cookbooks (`cluster_service_discovery`, motd, etc) and system cookbooks(hadoop, cassandra, etc) are housed in two separate folders.
- The standard layout would just include the cookbooks, but a cluster-oriented approach demands that the roles travel along too
* (possibly) `cluster_chef-chef-repo` (??better name, anyone??) -- a fork of https://github.com/opscode/chef-repo that integrates the above

#### Handling of cookbooks that originate from opscode-cookbooks

Right now we *copy* standard cookbooks from opscode's repo into the `cookbooks` directory. This lets us version them separately, but means we have to track them, and could cause conflicts with the majority of people who will be pulling from opscode-cookbooks already.

1. omit entirely, but list as dependencies (my vote)
2. `git subtree` pull them into the cluster_chef-cookbooks repo
3. copy them in as we've been doing
4. `git submodule` opscode-cookbooks and symlink

#### Organization of new repo(s)

Opscode recommends a [standard layout for your chef repo](https://github.com/opscode/chef-repo). We should make the new arrangement work seamlessly within that structure.

The new layout should
* easy to integrate if you have your own existing chef-repo
* straightforward for a new chef user to adopt
* either mirror or be what we actually use

Proposed, but this doesn't feel solid to me yet:

```
clusters/ # internal clusters
roles/ # internal roles
site-cookbooks/ # internal cookbooks

systems-cookbooks@ # symlink to cluster_chef-systems/systems-cookbooks
cloud-cookbooks@ # symlink to cluster_chef-systems/cloud-cookbooks
opscode-cookbooks/ # git submodule of https://github.com/opscode/cookbooks

cluster_chef-systems # git submodule of https://github.com/infochimps/cluster_chef-systems
clusters/
roles/
systems-cookbooks/ # hadoop, cassandra, etc
cloud-cookbooks/ # cluster_service_discovery, motd, etc

.chef/ # knife config, keypairs, etc
certificates/
config/
data_bags/
environments/
```

The idea is that you can either just include the cookbooks or clusters directories in your config file, or symlink selectively.