hashicorp/serf — pull request #194 by ryanuber (2014-03-29 02:10:14+00:00)
Title: feature: rotate encryption keys

Took a stab at one approach for swapping out the encryption key. It seems to be working so far on clusters size 1-99 nodes running locally.

The main problem is deciding when to swap the key. In this implementation, a small number of messages are likely to be rejected, since all members in the cluster don't necessarily swap keys at the same exact instant. This hasn't caused failed member issues for me yet but I could see how that could happen.

This pull is mainly to solicit some feedback and gather ideas on how best to support this functionality in Serf, which is why it has no tests/docs etc.

It's probably also possible to handle key rotation using queries / events with handlers and SIGHUP'ing the agent, but I think this would face the same problems. I do think we need a good, integrated way of rotating the key, especially if Serf is to be used in production environments.

Another thought would be to allow a `AltSecretKey` or similar in Memberlist config, and fall back to using that to decrypt messages if `SecretKey` fails. This would give us a little leniency in getting a new key in place. We could call `SwapKeys()` or something like that on memberlist, which would allow us to start encrypting with the new key, and fall back decrypting with the old one. Just an idea.

Here is the high-level approach I took:

1. User initiates key rotation using `serf rotate-key 8M1Sbtz6ElPdDHfDbzbNJg==`
2. Serf sends an `internal_query` to all members with the new key contained in the payload.
3. Each member receives the query and stores the new key in `serf.config.NewSecretKey`, then sends an ack.
4. Serf counts received acks vs. number of total members in the cluster. If any node fails to ack receiving the new key, then key rotation is aborted.
5. Serf broadcasts a `rotate-key` event to all members, indicating that they should now swap keys.
6. Each member receives `rotate-key`, and starts a new goroutine which will sleep for `serf.config.BroadcastTimeout` so that it has a chance to rebroadcast successfully before the key is swapped.
7. [in background] Sleep for duration of `serf.config.BroadcastTimeout` so that Serf has a chance to rebroadcast successfully before the key is swapped. Once the sleep has expired, the `serf.config.MemberlistConfig.SecretKey` gets swapped out.
8. Agent receives a `rotate-key` event containing the base64-encoded key as the payload, allowing the user to handle patching a config file.

Any and all thoughts/suggestions welcome!