pingles/clj-kafka — pull request #7 by DAddYE (2013-05-15 07:17:37+00:00)
Title: Prevent Out of Memory with large streams

This version caps the queue and prevent OOM and obviously uses much less memory.

My app that use some like:

(doseq [group (partition resolution (kafka/messages consumer topic))

Where resolution is a value of ~60000

Without this patch the memory usage was constantly around ~4gb

After dealing with OOMs and inspecting all vm snapshots I come that the problem was in the pipe, I checked your code then the linked article and was confirmed (in one comment) with the same solution.