bitly/nsq — pull request #429 by mreiferson (2014-08-02 14:32:35+00:00)
Title: nsqd: --worker-id issues

It appears there are some cases where nsqd will fail to FIN a message because it thinks it's not in flight (but the message is actually in flight).

In playing around with nsqd locally, I kept seeing messages like this in the log:

```
E_FIN_FAILED FIN 06dbb0680b436005 failed ID not in flight
```

This was happening with a single consumer on a topic that was doing nothing but pulling messages off the queue and immediately FIN'ing them.

At first I thought it was a bug with the client library I was writing, but I was able to reproduce it with other client libraries as well (krakow and pynsq).

After some trial and error, I've found a way to reliably reproduce it:
1. Start nsqd
2. Write messages to it quickly
3. Simultaneously, read and fin messages as fast as possible while using a reasonably high max_in_flight (50+)

Here's a small Ruby program that will reproduce it every time:
https://gist.github.com/bschwartz/d9b82670c48bcf6a5b7d

The output for it looks like this:
```
➜ nsq-fin-failer ruby fin-failer.rb
.
.
.
.
.
FIN Failed for: 4165
FIN Failed for: 4166
FIN Failed for: 4167
FIN Failed for: 4168
FIN Failed for: 4169
FIN Failed for: 4170
FIN Failed for: 4171
FIN Failed for: 4172
FIN Failed for: 4173
FIN Failed for: 4174
FIN Failed for: 4175
FIN Failed for: 4176
FIN Failed for: 4177
FIN Failed for: 4178
FIN Failed for: 4179
FIN Failed for: 4180
FIN Failed for: 4249
FIN Failed for: 4250
FIN Failed for: 4251
FIN Failed for: 4252
FIN Failed for: 4253
FIN Failed for: 4254
FIN Failed for: 4256
FIN Failed for: 4257
FIN Failed for: 4258
FIN Failed for: 4259
FIN Failed for: 4260
FIN Failed for: 4261
FIN Failed for: 4321
FIN Failed for: 4322
.
```

Hopefully it's clear what's going on in there. I am hoping to read through the source of nsqd and help track this down, but thought I'd throw it up here in case you guys know right away what might be causing it.

Thanks for building a great piece of software!