mumrah/kafka-python — pull request #36 by mahendra (2013-07-30 02:58:02+00:00)
Title: Avoid multi-threaded auto-commit

Note: This is done on top of #33

This is done using shared memory objects provided by multiprocessing.

Design
* We start a commit process which waits on a multiprocessing.Queue with timeout = auto_commit_every_t
* In the main proc, when _auto_commit triggers a commit, we send a message on the queue
* The commit proc wakes up and does a commit
* If timeout is not None, then the commit proc will wake up after the specified timeout and to the commit.
* Data is shared between both procs using shared memory.
* We avoid the use of Locks completely

Also fix a bug in auto-commit for - auto_commit_every_n

The diffs can be seen at :
https://github.com/mahendra/kafka-python/compare/partition...mpcommit