jtriley/StarCluster — pull request #321 by nkrumm (2013-11-09 21:39:00+00:00)
Title: ENH: Use DNS server on master to resolve hostnames (instead of /etc/hosts)

Hi all,

I'd love to get some early feedback on this enhancement, which replaces the time-consuming process of updating all /etc/hosts files on all the nodes every time a node is added or removed. In my experience, with large clusters (100+ nodes), this process can take hours. In this PR, I replace the reliance on /etc/hosts with a DNS a simple DNS server on the master node, and add a new "prepend domain-name-servers" entry to /etc/dhcp/dhclient.conf on all nodes. This bypasses the O((N^2)/2) operation required to update all the /etc/hosts on all nodes when adding or removing N nodes. Instead, only the /etc/hosts file on master must be updated.

A couple of clarifications/notes:

1) The /etc/hosts file must still contain the entry for the node itself, as SGE's /opt/sge6/utilbin/linux-x64/gethostname requires this (and SGE install will fail without this present)
2) Typically, one can specify a "nameserver" entry in /etc/resolv.conf, but because this file is frequently overwritten on AWS (when DHCP updates, for example), DNS nameserver settings need to be registered in the dhclient.conf file instead.

Other thoughts:

- Although this seems to work quite well in my hands, I'd love feedback on how to best implement the addition of the lines to /etc/dhcp/dhclient.conf. The way it works now (sed find/replace) seems brittle at best...
- The DNSmasq server could probably be included with the AMI and started by default on all images with no harm. But perhaps we would want this to be more configurable? I'm not terribly familiar with DNS and so don't know offhand what people would like to configure.
- What kind of issues would this run into with the VPC branch? I took a quick look and it seems as if StarCluster-VPC still makes use of the /etc/hosts arrangement, but I could be wrong.
- The option to use DNS should probably be a configurable option in the StarCluster config file, but i've left that out for now.
- Documentation: pending!
- Lastly, there are a few other lengthy operations done when adding or removing nodes that become very costly with large clusters (e.g., updating the total # of slots/cpus available in SGE). These can be bypassed in similar ways, but I left them out of this PR for now. In the future, i'll submit a "large cluster performance support" PR for those.

Thanks for reading, happy to take feedback!
Nik