airbnb/synapse — pull request #24 by jamessharp (2013-11-13 21:27:22+00:00)
Title: Load services dynamically

This is a bit of a speculative PR

I've been integrating nerve/synapse into a [mesos](http://mesos.apache.org) cluster. Essentially I want to setup a cluster that I can distribute a number of services across and have automatic TCP routing such that if I declare that a service listens on port A, I can visit port A on any server in the cluster and get routed to an instance of the service, wherever it happens to live.

Combining nerve and synapse seems like a good choice for doing this, however I ran into a bit of a problem - you have to configure the synapse services up front. This was less than ideal because I don't want to have to go and reconfigure each instance of synapse individually every time I introduce a new service. Ideally what would happen is that synapse discovers that a new service has been deployed, it goes and fetches the config from somewhere and reloads itself with the new service.

This is what this PR attempts to do.

I've introduced two new components - a Service Discoverer and a Config Fetcher (not my finest hour when it comes to inventing names…). The Service Discoverer is responsible for discovering which services are in play and should be routed to. The Config Fetcher is responsible for fetching the config for new/unknown services.

At the moment I've just got a zookeeper discoverer and an S3 fetcher (the config is placed in the S3) but it looks like it is working quite well, and others can be added at any time.

Here is an example of a new `config.json` to enable this functionality

{
"services": {}
"haproxy": {
...
},
"service_discoverer": {
"method": "zookeeper",
"hosts": [
"10.1.2.2:2181",
"10.1.2.3:2181",
"10.1.2.4:2181"
],
"path": "/smartstack/services",
"fetcher": {
"method": "s3",
"credentials": "/opt/smartstack/aws_credentials.yml",
"bucket": "service-config",
"default_discovery": {
"method": "zookeeper",
"hosts": [
"10.1.2.2:2181",
"10.1.2.3:2181",
"10.1.2.4:2181"
],
"path": "/smartstack/services/%SERVICE_NAME%/services"
}
}
}
}

and here is a config file that would be stored on the S3 and fetched

{
"synapse": {
"haproxy": {
"port": 4400,
"server_options": "check inter 2s rise 3 fall 2",
"listen": [
"mode http",
"option httpchk /ping"
]
}
},
"nerve": {
"check_interval": 1,
"checks": [
{
"type": "http",
"uri": "/ping",
"timeout": 1,
"rise": 1,
"fall": 1
}
]
}
}

I'm not sure this code is production ready yet, but would be interested to know whether you'd be interested in having something like this in the core code?