alphagov/govuk_crawler_worker — pull request #5 by dcarley (2014-07-01 08:06:03+00:00)
Title: Redis reconnect after error

The radix/redis client closes the connection to the server if it experiences
an error. This is what causes the error caught in 8c907f8. Improve our
reliability by reconnecting the client if we encounter a network or IO
related error (but not a semantic error like an invalid command).

This will not block the operation that first caused the error, meaning that
PublishURL() may submit one already crawled URL and ReadFromQueue() may
reject (requeue) an item.

The reconnect will run in another goroutine but subsequent operations will
be blocking, causing PublishURL() and ReadFromQueue() to then stop
processing until Redis is available again. This is a desirable result
because without the cache we'd expend resources re-crawling already
up-to-date pages.

There is a 2s delay between reconnect attempts so as not to overwhelm the
client or server. But no limit on how many times it will try to reconnect.

I'm unsure about how we're going to monitor this because the health check
endpoint which is going to use ttlHashSet.Ping() will also block and we
don't have any logging inside of ttlHashSet.

---

Is this the right approach? Open to suggestions about that last paragraph.