OpenTSDB/tcollector — pull request #75 by jtakkala (2013-04-09 15:26:50+00:00)
Title: Blacklist host when send fails, fixes error when only one TSD defined

This fixes an issue we had which occurs when one TSD is defined and becomes unavailable (eg. due to TSD restart), which causes tcollector to exit. If the TSD sends a TCP connection reset, then tcollector gets caught in a loop which it does not recover from, and eventually exits.

```
2013-03-07 10:39:27,445 tcollector[2613] ERROR: failed to send data: [Errno 32] Broken pipe
2013-03-07 10:39:28,465 tcollector[2613] INFO: No more healthy hosts, retry with previously blacklisted
2013-03-07 10:39:28,465 tcollector[2613] ERROR: Uncaught exception in SenderThread, ignoring
Traceback (most recent call last):
File "/usr/local/tcollector/tcollector.py", line 446, in run
self.maintain_conn()
File "/usr/local/tcollector/tcollector.py", line 576, in maintain_conn
self.pick_connection()
File "/usr/local/tcollector/tcollector.py", line 423, in pick_connection
self.host, self.port = self.hosts.pop(0)
IndexError: pop from empty list
[snip]
2013-03-07 10:43:40,838 tcollector[2613] INFO: shutting down children
[snip]
2013-03-07 10:43:48,846 tcollector[2613] INFO: exiting
```