Repository: Parsely/pykafka

Issue #661: Native consumer hits error backoff delay after topic EOF in end-to-end testing.
Created: 2017-03-14T16:30:26Z
Good day!
I have been testing end-to-end performance of pykafka, similar to what is done in the EndToEndLatency test.
I have been very surprised that using rdkafka often made the latency considerably higher. Here is my test harness:
from time import time
from random import random
from os import getpid

from pykafka import KafkaClient
from pykafka.common import OffsetType

TIMEOUT = 6000
TOPIC = 'test'
KEY = 'key'

client = KafkaClient(
    zookeeper_hosts='localhost:2181',
    broker_version='0.10.2',
    socket_timeout_ms=TIMEOUT)
topic = client.topics[TOPIC]


def produce_and_consume(key, value, producer, consumer):
    producer.produce(value, key)
    while True:
        t = time()
        msg = consumer.consume()
        delta = time() - t
        if msg.partition_key != key:
            continue
        if msg.value != value:
            continue
        return delta

if __name__ == '__main__':
    num_seen = 0
    pid = getpid()
    producer = topic.get_producer(
        True,
        sync=True,
        retry_backoff_ms=10,
        linger_ms=1)
    producer.start()
    for use_lib in (True, False):
        deltas = []
        consumer = topic.get_simple_consumer(
            "a_group", use_lib,
            auto_commit_enable=False,
            auto_offset_reset=OffsetType.LATEST,
            fetch_wait_max_ms=0,
            # fetch_error_backoff_ms=1,
            consumer_timeout_ms=TIMEOUT)

        for i in range(20):
            value = "%d_%d" % (pid, i)
            delta = produce_and_consume(KEY, value, producer, consumer)
            deltas.append(delta)
        print "lib=%d, n=%d, low=%f, high=%f, avg=%f" % (
            use_lib, i, min(deltas), max(deltas),
            sum(deltas) / len(deltas))
        consumer.commit_offsets()
        consumer.stop()
    producer.stop()
Here are typical results:
lib=1, n=19, low=0.499292, high=1.004887, avg=0.526834
lib=0, n=19, low=0.000024, high=0.009366, avg=0.001291

This was done with kafka 0.10.2, pykafka 2.5.0 or 2.6.0.dev2, on MacOS 10.12.3.
After some analysis, I realized that we were almost always going through the recursive use of Consumer_consume following a RD_KAFKA_RESP_ERR__PARTITION_EOF here
(I am not sure why it hits EOF with a synced topic; I tried with native or non-native producers, but it has no impact. On an unrelated note, I wonder why the Topic.get_synced_producer always gives a non-native producer, is that intentional?)
After some more research, I found that this is a known feature of rdkafka: EOF is counted as an error and triggers the error backoff. See
confluentinc/librdkafka#1024
(even though we are not in the enable.partition.eof = false case, the rest of the discussion is relevant.)
I tried setting a low fetch.error.backoff.ms by adding that parameter to pykafka (commented parameter in my harness), and I got perfectly fine results with the rdkafka in that way.
It is a temporary solution, until the librdkafka issue is fixed, but it is the first one I have to propose at this point.

Duplicate issue #665: Rd Kafka Async Producer slow
Created: 2017-03-16T23:18:49Z
Hey guys, I've been testing the async producer with and without rdkafka and it seems like the rdkafka version is slower. Any known reasons fro this?

Closing comment (2017-03-23T16:57:35Z): Closing this as a duplicate of #661. If you don't think that's accurate @kariya2, please comment.