bitly/nsq — pull request #390 by mreiferson (2014-07-10 08:29:56+00:00)
Title: nsqd: messages stuck in-flight

Had an issue running nsqd tip where a client locked up, got kicked due to IO timeout, and messages that were in-flight never timed out and continued to be reported as in-flight by nsqd. Unfortunately i wasn't able to investigate further because we flushed the channel, and the messages stuck in-flight timed out.

To be clear, there were no clients (they got disconnected due to io timeouts) and the messages stayed listed as in-flight (for 10+ minutes) beyond when they should have timed out. Other nsqd instances running an older version (0.2.23-alpha - d555c0eca575c5c93ff45f39cccadad10c5ad5f8) timed out messages when presented with the same issue from the same consumer.

Logs I do have all indicate all client connections cleaned up correctly. (all connections appeared similarly)

```
2014/06/28 02:55:07 TCP: new client(10.222.103.153:56563)
2014/06/28 02:55:07 CLIENT(10.222.103.153:56563): desired protocol magic ' V2'
....
2014/06/28 15:52:37 PROTOCOL(V2): [10.222.103.153:56563] exiting ioloop
2014/06/28 15:52:37 ERROR: client(10.222.103.153:56563) - read tcp 10.222.103.153:56563: i/o timeout
2014/06/28 15:52:37 PROTOCOL(V2): [10.222.103.153:56563] exiting messagePump
```