Package: afkak
Afkak is aTwisted-nativeApache Kafkaclient library.
It provides support for:Producing messages, with automatic batching and optional compression.Consuming messages, with group coordination and automatic commit.Learn more in thedocumentation, downloadfrom PyPI, or review thecontribution guidelines.
Please report any issueson GitHub.StatusAfkak supports these Pythons:CPython 3.5, 3.6, 3.7, 3.8, and 3.9PyPy3We aim to support Kafka 1.1.xand later.
Integration tests are run against these Kafka broker versions:0.9.0.11.1.1Testing against 2.0.0 is planned (see#45).Newer broker releases will generally function, but not all Afkak features will work on older brokers.
In particular, the coordinated consumer won’t work before Kafka 0.9.0.1.
We don’t recommend deploying such old releases of Kafka anyway, as they have serious bugs.UsageHigh levelNote: This code is not meant to be runnable. Seeproducer_exampleandconsumer_examplefor runnable example code.fromafkak.clientimportKafkaClientfromafkak.consumerimportConsumerfromafkak.producerimportProducerfromafkak.commonimport(OFFSET_EARLIEST,PRODUCER_ACK_ALL_REPLICAS,PRODUCER_ACK_LOCAL_WRITE)kClient=KafkaClient("localhost:9092")# To send messagesproducer=Producer(kClient)d1=producer.send_messages("my-topic",msgs=[b"some message"])d2=producer.send_messages("my-topic",msgs=[b"takes a list",b"of messages"])# To get confirmations/errors on the sends, add callbacks to the returned deferredsd1.addCallbacks(handleResponses,handleErrors)# To wait for acknowledgements# PRODUCER_ACK_LOCAL_WRITE : server will wait till the data is written to#                         a local log before sending response# [ the default ]# PRODUCER_ACK_ALL_REPLICAS : server will block until the message is committed#                            by all in sync replicas before sending a responseproducer=Producer(kClient,req_acks=Producer.PRODUCER_ACK_LOCAL_WRITE,ack_timeout=2000)responseD=producer.send_messages("my-topic",msgs=[b"message"])# Using twisted's @inlineCallbacks:responses=yieldresponseDifresponse:print(response[0].error)print(response[0].offset)# To send messages in batch: You can use a producer with any of the# partitioners for doing this. The following producer will collect# messages in batch and send them to Kafka after 20 messages are# collected or every 60 seconds (whichever comes first). You can# also batch by number of bytes.# Notes:# * If the producer dies before the messages are sent, the caller would# * not have had the callbacks called on the send_messages() returned# * deferreds, and so can retry.# * Calling producer.stop() before the messages are sent will# errback() the deferred(s) returned from the send_messages call(s)producer=Producer(kClient,batch_send=True,batch_send_every_n=20,batch_send_every_t=60)responseD1=producer.send_messages("my-topic",msgs=[b"message"])responseD2=producer.send_messages("my-topic",msgs=[b"message 2"])# To consume messages# define a function which takes a list of messages to process and# possibly returns a deferred which fires when the processing is# complete.defprocessor_func(consumer,messages):#  Store_Messages_In_Database may return a deferredresult=store_messages_in_database(messages)# record last processed messageconsumer.commit()returnresultthe_partition=3# Consume only from partition 3.consumer=Consumer(kClient,"my-topic",the_partition,processor_func)d=consumer.start(OFFSET_EARLIEST)# Start reading at earliest message# The deferred returned by consumer.start() will fire when an error# occurs that can't handled by the consumer, or when consumer.stop()# is calledyielddconsumer.stop()kClient.close()Keyed messagesfromafkak.clientimportKafkaClientfromafkak.producerimportProducerfromafkak.partitionerimportHashedPartitioner,RoundRobinPartitionerkafka=KafkaClient("localhost:9092")# Use the HashedPartitioner so that the producer will use the optional key# argument on send_messages()producer=Producer(kafka,partitioner_class=HashedPartitioner)producer.send_messages("my-topic","key1",[b"some message"])producer.send_messages("my-topic","key2",[b"this method"])Low levelfromafkak.clientimportKafkaClientkafka=KafkaClient("localhost:9092")req=ProduceRequest(topic="my-topic",partition=1,messages=[KafkaProtocol.encode_message(b"some message")])resps=afkak.send_produce_request(payloads=[req],fail_on_error=True)kafka.close()resps[0].topic# b"my-topic"resps[0].partition# 1resps[0].error# 0 (hopefully)resps[0].offset# offset of the first message sent in this requestInstallAfkak releases areavailable on PyPI.Because the Afkak dependenciesTwistedandpython-snappyhave binary extension modules you will need to install the Python development headers for the interpreter you wish to use:Debian/Ubuntu:sudo apt-get install build-essential python3-dev pypy3-dev libsnappy-devOS Xbrew install python pypy snappypip install virtualenvThen Afkak can beinstalled with pip as usual:LicenseCopyright 2013, 2014, 2015 David Arthur under Apache License, v2.0. SeeLICENSECopyright 2014, 2015 Cyan, Inc. under Apache License, v2.0. SeeLICENSECopyright 2015–2021 Ciena Corporation under Apache License, v2.0. SeeLICENSEThis project began as a port of thekafka-pythonlibrary to Twisted.SeeAUTHORS.mdfor the full contributor list.