Introduced a new docker container to the local cluster deployment. The new container uses a custom docker image to instatiate a DNS server via the somewhat lightweight dnsmasq utility (as opposed to bind). This DNS is then carefully fed the...

Summary:
...addresses of each Cockroach node container, so that they can all find each other via hostnames.

We could potentially have switched cockroach to dealing only with IP
addresses, but I've been informed that the hostname is an important part
of the TLS authentication, so that would probably be a mistake.

Introducing this DNS container was very difficult however. The big issue
is that you must create the DNS container first so that you have its
IP address for specification via docker --dns=$DNS_IP for each of the
Cockroach node containers. However, when you instantiate the DNS first,
you obviously don't yet have the correct IP addresses for the as-yet-not-
instatiated cockroach node containers. Chicken and egg problem and very
fucking obnoxious. I have no idea why Docker doesn't have some better
mechanism for all running containers to be able to resolve each other
by the --hostname you specify when you create the containers.

Further complicating things is the fact that boot2docker doesn't have
good support for sharing an OSX folder down through the virtual machine
and then into the DNS container. Instead, we're forced to move the file
we want to share with the DNS container (which contains the list of IP HOST
pairs for the cockroach container nodes) via SSH.

The DNS container is run such that it waits on a 1-second polling interval
for the appearance of an additional-hosts file for dnsmasq. Once the file
appears, dnsmasq is started and begins to serve dns entries for the named
nodes (we use node[0-9] as hostnames).

Some additional changes were necessary to the gossip bootstrap to not
throw away initially-unresolvable hosts. Instead, we attempt to re-resolve
all hosts in the gossip bootstrap hosts array when we can't connect. This
is probably the right thing to do anyway and allows our DNS entries to be
reconfigured on the fly and all nodes to eventually reconnect to the gossip
network should they become unmoored.

Frustratingly, this 65 line change took me about 10 hours. This stuff is
just not easy at all.

Test Plan: make acceptance

Reviewers: andybons, petermattis, bdarnell, Tobias

Reviewed By: Tobias

Subscribers: team

Differential Revision: http://phabricator.cockroachdb.org/D164