Docker Clustering: Design proposal.

Authors: @aluzzardi and @vieux.

The goal and scope of this change is to allow docker to manage a cluster
of docker hosts.

The target audience for clustering is modeled after Docker itself:
developer and devops first then enterprise later. Enterprise requires
additional features such as authentication, ACLs, auditing and tooling
which will come at a later time.

The system is designed to handle workloads in the same way as Docker: it
can run both long-running and one-off tasks. Batch processing can be
built on top of the API using the one-off primitive.

The architecture is based on an evented model where the master queries
the registered slaves within the system and checks the current state of
the slaves against the requested state by the user.  It reconciles any
differences and updates the cluster state with the required changes.

Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
Signed-off-by: Victor Vieux <vieux@docker.com>
I assume the leader election would be through an implementation of Raft as mentioned earlier today? I thin a gossip protocol like Serf might be useful for failure detection and state distribution to route important messages around failed nodes or network partitions.

-g
> On Oct 30, 2014, at 3:14 PM, Jessie Frazelle <notifications@github.com> wrote:
> 
> In docs/sources/userguide/cluster.md:
> 
> > +    $ docker -d --discovery https://discovery.hub.docker.com/u/demo-user/cluster-1
> > +
> > +
> > +In the future, other discovery mechanism will be provided that will not depend on the Hub.
> > +
> > +## Master
> > +
> > +In order to ensure consistency within the cluster one of your Docker Engines
> > +will need to be promoted to master within the cluster.  If you are using
> > +the Hub's discovery service you will be able to promote any of your registered
> > +nodes to a master from the web interface.  You can also statically assign one
> > +of the Docker Engines with the `--master` flag.
> > +
> > +    $ docker -d --master --discovery https://discovery.hub.docker.com/u/demo-user/cluster-1
> > +
> > +Eventually, clustering will provide a built-in leader election algorithm making the `--master` flag obsolete.
> Would this use a gossip protocol to assign a new master, probably out of scope for the initial proposal just wondering
> 
> —
> Reply to this email directly or view it on GitHub <https://github.com/docker/docker/pull/8859/files#r19629151>.
>