richardhenry/hotqueue — pull request #10 by j4mie (2012-04-12 14:43:41+00:00)
Title: Add bulk context manager, disable serializer

Hi Richard,

First - thanks for HotQueue. It's beautifully simple and totally bulletproof.

This pull request adds two new features which address particular pain points I've had, mostly around enqueuing very large number of items quickly. Both features are in one PR, which isn't ideal, but as they both touch some of the same code I couldn't find an easy way to open separate PRs which would both apply cleanly. If you like one of these features but not the other, please let me know and I'll create a new PR.

I've added docs and tests, and updated the changelog. Please let me know if I've missed anything.

## 1. Bulk context manager

Occasionally, I need to enqueue very large numbers of items (reprocessing/reindexing lots of data, for example). Previously, HotQueue made a round-trip to Redis for every item that was added to a queue, which is a lot of overhead. This commit adds a `bulk` context manager which buffers items in memory, flushing them to Redis when the buffer is full. The size of the buffer can be tuned to control the speed/reliability tradeoff. Example:

>>> queue = hotqueue.HotQueue('myqueue')
>>> with queue.bulk(1000):
>>> for item in my_items:
>>> queue.put(item)

In my tests, this can easily increase enqueuing speed for a large batch of items by 20x. A context manager seemed like the cleanest solution here, as we need to make sure we flush the last batch of items off the queue when we leave the `bulk` block.

## 2. Allow serialization to be disabled completely

Although it's often useful to be able to place arbitrary Python objects onto a queue, sometimes your queue messages are just strings (database IDs, for example). In that case, the overhead of a serializer is quite significant. Even using a custom `DummySerializer` with `dumps` and `loads` methods that just return whatever they are passed adds method-call overhead. This commit simply allows `None` to be explicitly passed as the serializer, and a couple of `if self.serializer:` checks to skip serialization/deserialization. The default serializer (if the argument is not passed) remains `pickle`, as before.

A `DummySerializer` seems to be about 50% faster than pickling and unpickling strings, and a `None` serializer seems to be about 10% faster than a `DummySerializer`. Not a huge speedup, but it's significant if you have millions of messages. Also, `serializer=None` is much less typing than defining an entire class.

*Technically* this is a backwards-incompatible API change, but only if you were explicitly writing `queue = HotQueue('myqueue', serializer=None)`, which seems rather unlikely. As you're still pre-1.0, this doesn't really matter much. Still, you may want to make this version 0.3.0 instead of 0.2.7.

Cheers