Merge pull request #12 from davegardnerisme/per-message-requeue

Per-message Requeue
writer: s/done/finish
writer: return error during connection
writer: close the connection if IDENTIFY returns an error response
writer: dont defer the transaction cleanup so it happens before we signal the waitgroup
writer: better logging/cleanup/comments
writer: fix potential deadlocks on cleanup

When Writer cleans up after a connection error
it is possible for there to be a race between the cleanup
process (which stops the goroutine responsible for receiving
on transactionChan) and new concurrent writes (which send to
transactionChan), resulting in potential deadlocks until
a new successful connection is made.

This is exemplified in nsq_to_nsq, where Writer is used
alongside Reader such that, after seeing an error writing,
Reader enters backoff mode and sends RDY 1...

Because a stray write would be blocked on sending
to transactionChan the source nsqd would (correctly) stop
sending additional messages (RDY 1 == in-flight 1).

This, coupled with the fact that Writer only lazily connects
to its destination hosts, meant that until that in-flight
message timed out at the source nsqd there would be no
additional message delivered to force a re-connect and
free up the deadlock.

This fix adds accounting for the # of concurrent writers
and ensures that transactionCleanup() recvs on transactionChan
until that count reaches 0.
Merge pull request #13 from mreiferson/writer_deadlock_13

writer: potential deadlocks on cleanup
reader: improve RDY handling

* consolidate RDY handling to a single goroutine per Reader
  instead of per connection.  this simplifies the code paths
  and eliminates the need to synchronize across connections.
  (ie. when you encounter backoff you want all of the connections
  for that Reader to backoff).

* when exiting backoff send RDY 1 to single, random, connection.

* improve redistribution to better detect and handle cases
  where disconnections need their orphaned RDY count
  transferred to existing connections, such as when in backoff.
reader: add mock tests for RDY handling
reader: cleanup handler code
reader: simplify exit timer code
reader: re-org

 * trim long lines
 * separate files for nsqConn and Reader
 * comment cleanup
reader: synchronize cleanup of connection
reader: guarantee cleanup of in flight messages
reader: cleanup lookupd query index code
reader: synchronize access to shared lookupd slice
reader: fix race in max backoff count
Merge pull request #8 from mreiferson/tests_8

reader: tests / centralize RDY handling
bump stable 0.3.3