elasticsearch/elasticsearch-river-rabbitmq — pull request #38 by zoomix (2013-09-11 16:13:29+00:00)
Title: Allow setting QoS parameters like prefetch size and count.

Added ability to configure pre-fetch parameter of rabbit-mq and set pre-fetch count to 10 as default.

Why is it needed in the first place:
Without pre-fetch parameters the rabbit-mq river will attempt to download all messages locally on the elastic search instance. So if you have a large amount of messages on queue the elastic search instance will OOM itself to death just trying to be clever and pre-fetching everything (which is the default behavior of the rabbit-mq client). Restarting won't do much good since it's only going to OOM again.. In poetic terms: the river's gonna drown the elastics.

10 as default:
It's a value we've arrived at after couple of years of using rabbit-mq on EC2 for all sorts of purposes. 10 protects you from losing messages and drowning instances. Increasing it seldom gives any meaningful performance benefits. If one seriously needs performance it is far better to batch and pre-compile messages before putting them on the mq.