python/psf-salt — pull request #16 by dstufft (2014-10-26 19:35:01+00:00)
Title: Implement Service Discovery using Consul

* Sets up a consul server cluster (1 node in vagrant, 3 nodes in production).
* Installs a consul client agent on all other IAD nodes.
* Secures Consul
* Uses a random DC wide encryption key for encrypting the gossip protocol.
* Uses generates TLS certificates on the consul servers to ensure that RPC communcations are authenticated/encrypted.
* Registers the ``docs``, ``downloads``, ``hg``, and ``hg-ssh`` internal services.
* Registers the ``console`` external service pointing at pythonanywhere.
* Sets up HAProxy to use Consul instead of Salt Mine to discover services.
* Sets up consul-template to watch for changes and re-render the HAProxy configuration file and reload HAProxy.

So the major change here is that we switch from using Salt Mine to using Consul for service discovery. This has a number of benefits to us:

* No longer tied to only discovering nodes that are running salt, allowing external services (like pythonanywhere) to not be "special".
* Consul includes port information, so we no longer need to hard code ports.
* Consul includes health checks, which means we can (although not currently utilizing) register health checks for various services and nodes will automatically be removed from rotation if they fail health checks.
* Consul can react way faster than Salt, opening up the path to having auto-scaled services.
* Consul can be used for leader election, this allows Leader/Follower type services (PostgreSQL etc) to automatically select the leader in a safe way.
* Consul can be used as a cluster wide key/value store with ACLs and the like.
* This could be integrated with Salt itself so that salt can populate pillar values from Consul, removing the need to store generated values on the salt master file system and making the salt master easier to replace.

I also looked at Zookeeper, I know that @ewdurbin is a fan of ZK. I ultimately went with Consul over ZK because ZK itself is jsut a consistent key/value store. Adding anything on top of that requires other components or custom coding. While that gives a lot more flexibility that also means that there's a lot more moving pieces and places to go wrong. I've talked to some folks who are running large consul clusters in production and they are quite happy with it and I like the batteries includes service discovery of Consul. ZK is also written in Java which I'm kind of down on while Consul is written in Go and distributed as a single binary. The benefit of ZK would be that it's an older system that is more battle tested and understood.