hashicorp/consul — pull request #52 by carlosdp (2014-04-18 01:40:34+00:00)
Title: Failure lockout when starting up Agent using two different bind addresses

This is a weird case. If you start up an agent with one bind address, say:

```bash
consul agent -server -bootstrap -data-dir=/tmp/consul -bind=107.20.20.10
```

and then gracefully shutdown that agent and attempt to bring it back up with a different bind IP (say, as in my case, you made an error or didn't specify a bind address originally):

```bash
consul agent -server -bootstrap -data-dir=/tmp/consul -bind=127.0.0.1
```

The agent goes into a spin throwing RPC errors:

```bash
2014/04/17 20:52:17 [ERR] agent: failed to sync remote state: rpc error: No cluster leader
2014/04/17 20:52:17 [ERR] agent: failed to sync remote state: rpc error: No cluster leader
2014/04/17 20:52:17 [WARN] raft: Heartbeat timeout reached, starting election
2014/04/17 20:52:17 [INFO] raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state
2014/04/17 20:52:17 [ERR] raft: Failed to make RequestVote RPC to 192.168.0.191:8300: dial tcp 192.168.0.191:8300: connection refused
2014/04/17 20:52:18 [WARN] raft: Election timeout reached, restarting election
2014/04/17 20:52:18 [INFO] raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state
2014/04/17 20:52:18 [ERR] raft: Failed to make RequestVote RPC to 192.168.0.191:8300: dial tcp 192.168.0.191:8300: connection refused
```

and will not accept connections.

After looking through the code a little bit, this seems to be related to how peer list persistence is handled. Here, both 172.20.20.10 and 127.0.0.1 exist on the peer list upon boot which causes Raft to try to connect to itself twice in it's Candidate process (and I guess seek other local addresses to try). I'm actually not entirely sure if this is a Raft thing or Consul, but I ran into it here.

Quick Fix: Delete the `data-dir` to eliminate the old peer list if this mistake is made.

One immediate solution I thought of and am playing with locally is to remove any extra instances of localhost addresses from the peer list around `consul/server.go:270`, where the localhost is ensured to be entered into the list. This works when tried manually, but I bet there's a more elegant solution and I am just not familiar enough with this source yet to identify it.