spacemonkeygo/monitor — pull request #6 by pteichman (2014-11-07 20:42:26+00:00)
Title: Make ScribeCollector reconnect on write errors

Fixes #5

Apologies for the near-complete rewrite of ScribeCollector, but this matched up with some reconnection code we're using elsewhere that's pretty well tested. No API has been modified as part of this patch.

This introduces a buffered channel (`ScribeCollector.logs`) for collecting spans and a background goroutine that maintains a connection to Scribe for sending them out. Next steps could include coalescing multiple spans into a single Scribe Log.

In a failure situation, it will lose the first span it can't write to Scribe, and it will lose spans that arrive after its internal buffer has filled. That seems acceptable to me but you may feel differently.

This has now seen production use on my end, including killing scribe. Let me know if you'd like any changes!