Adding Requeue() method to messages for fine-grained control on requeue delays on message-by-message basis
Merge pull request #12 from davegardnerisme/per-message-requeue

Per-message Requeue
writer: s/done/finish
writer: return error during connection
writer: close the connection if IDENTIFY returns an error response
writer: dont defer the transaction cleanup so it happens before we signal the waitgroup
writer: better logging/cleanup/comments
writer: fix potential deadlocks on cleanup

When Writer cleans up after a connection error
it is possible for there to be a race between the cleanup
process (which stops the goroutine responsible for receiving
on transactionChan) and new concurrent writes (which send to
transactionChan), resulting in potential deadlocks until
a new successful connection is made.

This is exemplified in nsq_to_nsq, where Writer is used
alongside Reader such that, after seeing an error writing,
Reader enters backoff mode and sends RDY 1...

Because a stray write would be blocked on sending
to transactionChan the source nsqd would (correctly) stop
sending additional messages (RDY 1 == in-flight 1).

This, coupled with the fact that Writer only lazily connects
to its destination hosts, meant that until that in-flight
message timed out at the source nsqd there would be no
additional message delivered to force a re-connect and
free up the deadlock.

This fix adds accounting for the # of concurrent writers
and ensures that transactionCleanup() recvs on transactionChan
until that count reaches 0.
Merge pull request #13 from mreiferson/writer_deadlock_13

writer: potential deadlocks on cleanup