infochimps/cluster_chef — Issue #60: Complex hadoop cluster convergence should be more reliable
Author: mrflip
Created: 2011-10-23 19:49:14+00:00
State: open

Cluster convergence of complex hadoop clusters needs to be more solid.

While it would be awesome to have fire-and-forget for a full cluster, orchestrating the convergence and invocation of nearly a dozen services across as many as four supervisory machines is demostrably non-trivial. I'm at peace with a workflow that went something like
* launch namenode, wait for initial chef-client run to complete
* single-command script to format namenode and start services
* launch all remaining nodes
... and from there, evaluate full fire-and forget

----

### TODO

* Separate initial chef convergence of configuration from hadoop service launch
- Cluster recipes give you control over whether to *launch+enable*, *enable*, etc services by default.
- a single-step mechanism to switch the services on
- take another look at the namenode formatting step
* compatibility with CDH3u1 (which dicks around with a few paths).
* cluster_service provider racing
- When you stop-start machines, or otherwise need to reconverge, this will happily sail through with no notion of the race condition. Add a `knife cluster unprovide` that unregisters to provide all services, and have `knife cluster stop` invoke it along the way.
- add a `wait_for_provider` helper (or option) -- see fiksu@b65a7a5b74402c0c222ae9da27bd8163045997e8
* checkpoints so that if a service provider is missing (eg namenode not running) it doesn't cause chef to bomb out
* cluster reconvergence when machines are stopped and then started