hashicorp/serf — pull request #126 by ryanuber (2014-02-08 23:51:40+00:00)
Title: Change tags while Serf agent is running

Yo Hashicorp!

I was thinking of different use cases for Serf this weekend. I think I might have come across something useful for dynamic growing/shrinking of cluster services.

Two use cases in particular which brought my thinking here:

- Orchestration of database replication setup
- Dynamically adding/removing web nodes from a load balancer farm during a rolling upgrade

For these things to work in Serf how it is currently, there would likely be agent stopping/starting/config changes involved.

We might want for example, a new database node to come online, discover its peers and begin sucking down data from the other nodes in the cluster (replication), and then when replication is done, set a tag for itself to notify other members that it is ready for action.

Similarly for rolling upgrades, we might be adding tags to nodes to indicate which version(s) of an application they are serving, and using the version tags to configure different load balancing clusters, so once application version B is installed and running, it might make sense to just add a tag to the node to indicate it can be a member of the "app-B" cluster now.

I went a little overboard and attempted an implementation out of curiosity.

The patches here add:

- A `serf tags [-set tag1=val1] [-delete tag2]` command
- A TagsEvent so that we can persist tag changes in snapshot files

This would make the tagging system a bit more dynamic, while still allowing it to work the way it did before with the `serf agent -tag x=y` syntax. I think it would be nice to not have to modify the config file and restart the agent to change a tag value.

Anyways, let me know if you guys think this is a good idea, or if you have any other ideas for improvement/other ways to accomplish something like this, etc.