mumrah/kafka-python — pull request #29 by mahendra (2013-06-03 15:05:01+00:00)
Title: Fix auto-commit issues with multi-threading

This issue fixes the auto-commit threading issue that is there in the Kafka library. I thought of multiple approaches, but this simple approach seems to solve the problem.

## Design considerations
* The solution must not involve locking, since this can impact performance
* The solution must work in the fastest manner when threads are not used (for commit). For eg: adding a lock will be a detriment when auto_commit_every_t is set to None and the code is essentially single threaded.
* The code should not be overtly complex.

## Approaches considered
* Using a thread safe data structure like a dictionary to match correlation_id to the data
* To avoid thread issues, we can use an Event() to notify the thread of the data when it is available.
* Use a separate thread/process to send and fetch the data (way too complex).

However the base problem is that the recv operation is not atomic for a message. We first fetch 4 bytes and check the size of the message. We then fetch the remaining message based on this. This code is not re-entrant. Also reading the message is not in a single request. It is possible that both threads can read different bits of a message.

At this point, the simplest solution seems to allow both the threads to use their own sockets. So, we can get into creating multiple sockets etc.

A simpler solution to achieve this is to make the KafkaConnection object a thread-local object. This way, it is initialized for each thread that is created.