bitly/pynsq — pull request #34 by jehiah (2013-06-21 03:26:49+00:00)
Title: Error closing stale connections.

There is an error cleaning up from stale connections that's caused by updating the connection dictionary while iterating it. This is reproducible by doing blocking work in a task for longer than the heartbeat interval. After this state a new connection will be created, but NSQ will still have the old connection in the dictionary and will never send RDY count properly beyond the first message.

```
[W 130620 15:19:09 Reader:530] [nsqrelay10:4150:task] connection is stale (60.33s), closing
[W 130620 15:19:09 Reader:484] [nsqrelay10:4150:task] connection closed for task 'task'
[E 130620 15:19:09 ioloop:419] Error in periodic callback
Traceback (most recent call last):
File "/usr/local/lib/python2.5/site-packages/tornado/ioloop.py", line 415, in _run
self.callback()
File "/usr/local/lib/python2.5/site-packages/nsq/Reader.py", line 525, in check_last_recv_timestamps
for conn_id, conn in self.conns.iteritems():
RuntimeError: dictionary changed size during iteration
```

This is caused by `_close_callback` modifying the connection structure while iterating it.

cc: @mreiferson