bitly/go-nsq — pull request #13 by mreiferson (2013-10-06 04:54:26+00:00)
Title: writer: potential deadlocks on cleanup

When `Writer` cleans up after a connection error it is possible for there to be a race between the cleanup process (which stops the goroutine responsible for receiving
on `transactionChan`) and new concurrent writes (which send to `transactionChan`), resulting in potential deadlocks until a new successful connection is made.

This is particularly problematic for `nsq_to_nsq` where `Writer` is used alongside `Reader`. After seeing an error writing, `Reader` enters backoff mode and sends `RDY 1`. Because a stray write is blocked on sending to `transactionChan` the source `nsqd` would (correctly) stop sending additional messages (`RDY 1 == in-flight 1`).

This, coupled with the fact that `Writer` only *lazily* connects to its destination hosts, means that until the in-flight message times out at the source there would be no additional message delivered to force a re-connect and free up the deadlock.

This fix adds accounting for the # of concurrent writers and ensures that `transactionCleanup()` recvs on `transactionChan` until that count reaches `0`.

RFR @jphines @jehiah

I'll take payment in :beers: