Package: pykafka
PyKafkaPyKafka is a programmer-friendly Kafka client for Python. It includes Python
implementations of Kafka producers and consumers, which are optionally backed
by a C extension built onlibrdkafka. It runs under Python 2.7+, Python 3.4+,
and PyPy, and supports versions of Kafka 0.8.2 and newer.PyKafka’s primary goal is to provide a similar level of abstraction to theJVM Kafka clientusing idioms familiar to Python programmers and exposing
the most Pythonic API possible.You can install PyKafka from PyPI with$ pip install pykafkaor from conda-forge with$ conda install -c conda-forge pykafkaFull documentation and usage examples for PyKafka can be found onreadthedocs.You can install PyKafka for local development and testing by cloning this repository and
running$ python setup.py developGetting StartedAssuming you have at least one Kafka instance running on localhost, you can use PyKafka
to connect to it.>>>frompykafkaimportKafkaClient>>>client=KafkaClient(hosts="127.0.0.1:9092,127.0.0.1:9093,...")Or, for a TLS connection, you might write (and also seeSslConfigdocs
for further details):>>>frompykafkaimportKafkaClient,SslConfig>>>config=SslConfig(cafile='/your/ca.cert',...certfile='/your/client.cert',# optional...keyfile='/your/client.key',# optional...password='unlock my client key please')# optional>>>client=KafkaClient(hosts="127.0.0.1:<ssl-port>,...",...ssl_config=config)If the cluster you’ve connected to has any topics defined on it, you can list
them with:>>>client.topics>>>topic=client.topics['my.test']Once you’ve got aTopic, you can create aProducerfor it and start
producing messages.>>>withtopic.get_sync_producer()asproducer:...foriinrange(4):...producer.produce('test message '+str(i**2))The example above would produce to kafka synchronously - the call only
returns after we have confirmation that the message made it to the cluster.To achieve higher throughput, we recommend using theProducerin
asynchronous mode, so thatproduce()calls will return immediately and the
producer may opt to send messages in larger batches. TheProducercollects
produced messages in an internal queue forlinger_msbefore sending each batch.
This delay can be removed or changed at the expense of efficiency withlinger_ms,min_queued_messages, and other keyword arguments (seereadthedocs). You can still obtain
delivery confirmation for messages, through a queue interface which can be
enabled by settingdelivery_reports=True.  Here’s a rough usage example:>>>withtopic.get_producer(delivery_reports=True)asproducer:...count=0...whileTrue:...count+=1...producer.produce('test msg',partition_key='{}'.format(count))...ifcount%10**5==0:# adjust this or bring lots of RAM ;)...whileTrue:...try:...msg,exc=producer.get_delivery_report(block=False)...ifexcisnotNone:...print'Failed to deliver msg{}:{}'.format(...msg.partition_key,repr(exc))...else:...print'Successfully delivered msg{}'.format(...msg.partition_key)...exceptQueue.Empty:...breakNote that the delivery report queue is thread-local: it will only serve reports
for messages which were produced from the current thread. Also, if you’re usingdelivery_reports=True, failing to consume the delivery report queue will cause
PyKafka’s memory usage to grow unbounded.You can also consume messages from this topic using aConsumerinstance.>>>consumer=topic.get_simple_consumer()>>>formessageinconsumer:...ifmessageisnotNone:...printmessage.offset,message.value0testmessage01testmessage12testmessage43testmessage9ThisSimpleConsumerdoesn’t scale - if you have twoSimpleConsumersconsuming the same topic, they will receive duplicate messages. To get around
this, you can use theBalancedConsumer.>>>balanced_consumer=topic.get_balanced_consumer(...consumer_group='testgroup',...auto_commit_enable=True,...zookeeper_connect='myZkClusterNode1.com:2181,myZkClusterNode2.com:2181/myZkChroot'...)You can have as manyBalancedConsumerinstances consuming a topic as that
topic has partitions. If they are all connected to the same zookeeper instance,
they will communicate with it to automatically balance the partitions between
themselves. The partition assignment strategy used by theBalancedConsumeris
the “range” strategy by default. The strategy is switchable via themembership_protocolkeyword argument, and can be either an object exposed bypykafka.membershipprotocolor
a custom instance ofpykafka.membershipprotocol.GroupMembershipProtocol.You can also use the Kafka 0.9 Group Membership API with themanagedkeyword argument onget_balanced_consumer.Using the librdkafka extensionPyKafka includes a C extension that makes use of librdkafka to speed up producer
and consumer operation. To use the librdkafka extension, you need to make sure the header
files and shared library are somewhere where python can find them, both when you build
the extension (which is taken care of bysetup.py develop) and at run time.
Typically, this means that you need to either install librdkafka in a place
conventional for your system, or declareC_INCLUDE_PATH,LIBRARY_PATH,
andLD_LIBRARY_PATHin your shell environment to point to the installation location
of the librdkafka shared objects. You can find this location withlocate librdkafka.so.After that, all that’s needed is that you pass an extra parameteruse_rdkafka=Truetotopic.get_producer(),topic.get_simple_consumer(), ortopic.get_balanced_consumer().  Note
that some configuration options may have different optimal values; it may be
worthwhile to consult librdkafka’sconfiguration notesfor this.Operational ToolsPyKafka includes a small collection ofCLI toolsthat can help with common tasks
related to the administration of a Kafka cluster, including offset and lag monitoring and
topic inspection. The full, up-to-date interface for these tools can be fould by running$pythoncli/kafka_tools.py--helpor after installing PyKafka via setuptools or pip:$kafka-tools--helpPyKafka or kafka-python?These are two different projects.
Seethe discussion herefor comparisons
between the two projects.ContributingIf you’re interested in contributing code to PyKafka, a good place to start is the“help wanted”issue tag. We also recommend taking a look at thecontribution guide.SupportIf you need help using PyKafka, there are a bunch of resources available.
For usage questions or common recipes, check out theStackOverflow tag.
TheGoogle Groupcan be useful for more in-depth questions or inquries
you’d like to send directly to the PyKafka maintainers. If you believe you’ve
found a bug in PyKafka, please open agithub issueafter reading thecontribution guide.