ha/doozerd — Issue #19: A new Doozerd fails to join a cluster
Author: dannycoates
Created: 2012-01-19 04:49:08+00:00
State: open

Using the [firedrill](https://github.com/ha/doozerd/blob/master/doc/firedrill.md) and master branch as an example, if step 4 (start a forth) is run after the store revision number has passed the `-hist` parameter, the new instance panics with `TOO_LATE`

It's also easy to reproduce by setting a low `-hist` on the first instance and waiting to start another until the pulse has made rev > hist.

The stack trace begins:

```
panic: TOO_LATE

goroutine 1 [running]:
doozer/peer.cloner.VisitFile(0xf840051410, 0xf840001960, 0xf840019d60, 0xa, 0xf84004e3f0, ...)
/home/dcoates/code/doozerd/src/pkg/peer/peer.go:298 +0xc0
doozer/peer.(*cloner).VisitFile(0xf840019a60, 0xf840019d60, 0xf80000000a, 0xf84004e3f0, 0x1, ...)
/home/dcoates/code/doozerd/src/pkg/peer/peer.go:0 +0x8b
github.com/ha/doozer.walk(0xf840001960, 0x8f, 0xf840019d60, 0xf80000000a, 0xf84004e3f0, ...)
/home/dcoates/code/doozer/walk.go:23 +0x64
```

It appears that the `Clean` in clean.go (line 352) on the store initiated by a goroutine in peer.go (line 62) is causing the `Get` in peer.go (line 296) to return `ErrTooLate` because the rev of some path (in my case `/ctl/cal/0` at rev 5) is less than `st.head`

This seems like a bug, but I may be misunderstanding.

What is the proper way to add a new doozerd to a cluster after the rev has passed hist? This seems like a pretty basic operation to do if an instance crashes.