alphagov/content-store — pull request #52 by rboulton (2014-09-29 12:18:58+00:00)
Title: Enable publisher confirms when writing to rabbit

We want to make sure that things listening for publish events via a
rabbit queue get notified of all publish events. However, without
publisher confirms (or transactions, which are much more expensive in
terms of latency), there is no way to tell whether Rabbit has received a
message and placed it on the appropriate queues. This patch enables
publisher confirm mode on the channel used to publish messages to
rabbit, and waits for a confirmation for each message.

If no confirmation comes back (or a "nack" comes back indicating an
internal problem in Rabbit), this simply logs the error and the message
to errbit. Our usual action in this situation would be to republish
manually either the failed message or all documents modified since a
given timestamp.

Usually, it is desirable to either check for publisher confirms
asynchronously, or to batch checks, because it can take a while for
rabbit to send the ack (particularly if durable and persistent queues
are being used, because then the message needs to be written to disk
first). However, for our current expected use, waiting for each message
directly seems reasonable. If this has unacceptable performance, we can
investigate directly.

It is hard to test this patch; I've tested that the Bunny method gets
called, and I've checked that the connection from publisher to Rabbit is
in publish confirm mode by running the system on my dev VM and
inspecting the connection in the Rabbit console. I'm not sure how to
build an integration test of this, though, because getting nacks after
publish only happens when Rabbit is failing, and this is hard to
trigger.

For more information on publisher confirms, see
https://www.rabbitmq.com/confirms.html; for information on Bunny's
support, see
http://rubybunny.info/articles/extensions.html#publisher_confirms_publisher_acknowledgements