hashicorp/serf — pull request #199 by ryanuber (2014-04-11 08:13:33+00:00)
Title: feature: change encryption keys

For a week or so I've been chipping away at this feature little by little. It uses the new memberlist keyring feature to allow Serf to handle encryption key changes in a running cluster. I feel like it's feature-complete, and ready to start gathering some suggestions on the code itself.

There is more conversation around the feature in general in #194.

The bottom line on what it does can be seen with a few command examples. All of the operations are idempotent, so you can just keep running them without negative consequences if they fail.

Install a new key
```
$ serf key -install E4SoXyZSJkcg37n42U2i2Q==
==> Installing key on all members...
==> Successfully installed key!
```

Change the key used to encrypt messages:
```
$ serf key -use E4SoXyZSJkcg37n42U2i2Q==
==> Changing primary key on all members...
==> Successfully changed primary key!
```

List keys in use on the cluster:
```
$ serf key -list
==> Asking all members for installed keys...
==> Keys gathered, listing cluster keys...

E4SoXyZSJkcg37n42U2i2Q==
9kHC3w5eswavy1iU56YG8g==
```

Remove a key:
```
$ serf key -remove 9kHC3w5eswavy1iU56YG8g==
==> Removing key on all members...
==> Successfully removed key!
```

Error conditions during key operations:
```
$ serf key -remove E4SoXyZSJkcg37n42U2i2Q==
==> Removing key on all members...
failed: db9 Removing the primary key is not allowed
failed: web15 Removing the primary key is not allowed
failed: db8 Removing the primary key is not allowed
failed: web21 Removing the primary key is not allowed
failed: lb31 Removing the primary key is not allowed
failed: lb3 Removing the primary key is not allowed

Error removing key: 6/102 nodes reported failure
```

I know this pull request is pretty massive, so if splitting it into smaller chunks is preferred, I can try doing that. Just let me know!