bloomberg/chef-bcpc — pull request #88 by cbaenziger (2014-02-24 00:37:25+00:00)
Title: Update cluster-assign-roles for OpenStack or Hadoop install type

This update to cluster-assign-roles.sh provides for a more configurable Chef experience. Instead of requiring all nodes to fit into a model of being uniform head nodes or worker nodes, the cluster.txt now allows arbitrary role assignment (e.g. one node could be installed with multiple roles) and allows for pluggable installation types.

The reason to extend allowing pluggable installation types is due to a philosophical difference in how the original OpenStack BCPC code and newer Hadoop BCPC code was written to support cluster installs.

Graphically it can be seen (as red to green being first to latest Chef run):
![Uploading Cluster_Chefing.png . . .]()

In the OpenStack case, all headnodes are incrementally created. As each headnode is created, all future headnode's will know about it but all previous headnodes will be ignorant. As such, to install a cluster of headnodes, one runs an entire full Chef run from first to last headnode and then re-Chef and full Chef run last to first headnode. Daemons which need to know about future headnodes will thus need a bounce on a re-Chefing; luckily for our OpenStack clusters, this works fine as there's little need for a priori knowledge of the cluster layout or for daemons to be bounced on a re-Chefing.

For the Hadoop case, all headnodes are created by running a trivial bootstrap to create the Chef client and node object on the Chef server and create the necessary keys on the client. Then, we can go ahead and assign roles to all headnodes, on the Chef server. Lastly, we perform the full Chef run only once and all daemons can know where other Hadoop services will be on the cluster (even should they not yet be up). This requires one to install the cluster in order of the required daemon boot order (e.g. install HDFS and ensure it is started before installing HBase, which needs HDFS to be up and established). However, this allows us to install components with full a priori knowledge of the cluster layout.