Package: pup-confluent-kafka
Confluent's Python Client for Apache KafkaTMconfluent-kafka-pythonprovides a high-level Producer, Consumer and AdminClient compatible with allApache KafkaTMbrokers >= v0.8,Confluent Cloudand theConfluent Platform. The client is:Reliable- It's a wrapper aroundlibrdkafka(provided automatically via binary wheels) which is widely deployed in a diverse set of production scenarios. It's tested usingthe same set of system testsas the Java clientand more. It's supported byConfluent.Performant- Performance is a key design consideration. Maximum throughput is on par with the Java client for larger message sizes (where the overhead of the Python interpreter has less impact). Latency is on par with the Java client.Future proof- Confluent, founded by the
creators of Kafka, is building astreaming platformwith Apache Kafka at its core. It's high priority for us that client features keep
pace with core Apache Kafka and components of theConfluent Platform.See theAPI documentationfor more info.UsageBelow are some examples of typical usage. For more examples, see theexamplesdirectory or theconfluentinc/examplesgithub repo for aConfluent Cloudexample.Producerfromconfluent_kafkaimportProducerp=Producer({'bootstrap.servers':'mybroker1,mybroker2'})defdelivery_report(err,msg):""" Called once for each message produced to indicate delivery result.Triggered by poll() or flush(). """iferrisnotNone:print('Message delivery failed:{}'.format(err))else:print('Message delivered to{}[{}]'.format(msg.topic(),msg.partition()))fordatainsome_data_source:# Trigger any available delivery report callbacks from previous produce() callsp.poll(0)# Asynchronously produce a message, the delivery report callback# will be triggered from poll() above, or flush() below, when the message has# been successfully delivered or failed permanently.p.produce('mytopic',data.encode('utf-8'),callback=delivery_report)# Wait for any outstanding messages to be delivered and delivery report# callbacks to be triggered.p.flush()High-level Consumerfromconfluent_kafkaimportConsumerc=Consumer({'bootstrap.servers':'mybroker','group.id':'mygroup','auto.offset.reset':'earliest'})c.subscribe(['mytopic'])whileTrue:msg=c.poll(1.0)ifmsgisNone:continueifmsg.error():print("Consumer error:{}".format(msg.error()))continueprint('Received message:{}'.format(msg.value().decode('utf-8')))c.close()AvroProducerfromconfluent_kafkaimportavrofromconfluent_kafka.avroimportAvroProducervalue_schema_str="""{"namespace": "my.test","name": "value","type": "record","fields" : [{"name" : "name","type" : "string"}]}"""key_schema_str="""{"namespace": "my.test","name": "key","type": "record","fields" : [{"name" : "name","type" : "string"}]}"""value_schema=avro.loads(value_schema_str)key_schema=avro.loads(key_schema_str)value={"name":"Value"}key={"name":"Key"}defdelivery_report(err,msg):""" Called once for each message produced to indicate delivery result.Triggered by poll() or flush(). """iferrisnotNone:print('Message delivery failed:{}'.format(err))else:print('Message delivered to{}[{}]'.format(msg.topic(),msg.partition()))avroProducer=AvroProducer({'bootstrap.servers':'mybroker,mybroker2','on_delivery':delivery_report,'schema.registry.url':'http://schema_registry_host:port'},default_key_schema=key_schema,default_value_schema=value_schema)avroProducer.produce(topic='my_topic',value=value,key=key)avroProducer.flush()AvroConsumerfromconfluent_kafka.avroimportAvroConsumerfromconfluent_kafka.avro.serializerimportSerializerErrorc=AvroConsumer({'bootstrap.servers':'mybroker,mybroker2','group.id':'groupid','schema.registry.url':'http://127.0.0.1:8081'})c.subscribe(['my_topic'])whileTrue:try:msg=c.poll(10)exceptSerializerErrorase:print("Message deserialization failed for{}:{}".format(msg,e))breakifmsgisNone:continueifmsg.error():print("AvroConsumer error:{}".format(msg.error()))continueprint(msg.value())c.close()AdminClientCreate topics:fromconfluent_kafka.adminimportAdminClient,NewTopica=AdminClient({'bootstrap.servers':'mybroker'})new_topics=[NewTopic(topic,num_partitions=3,replication_factor=1)fortopicin["topic1","topic2"]]# Note: In a multi-cluster production scenario, it is more typical to use a replication_factor of 3 for durability.# Call create_topics to asynchronously create topics. A dict# of <topic,future> is returned.fs=a.create_topics(new_topics)# Wait for each operation to finish.fortopic,finfs.items():try:f.result()# The result itself is Noneprint("Topic{}created".format(topic))exceptExceptionase:print("Failed to create topic{}:{}".format(topic,e))Thread SafetyTheProducer,ConsumerandAdminClientare all thread safe.InstallInstall self-contained binary wheels$ pip install confluent-kafkaNOTE:The pre-built Linux wheels do NOT contain SASL Kerberos/GSSAPI support.
If you need SASL Kerberos/GSSAPI support you must install librdkafka and
its dependencies using the repositories below and then build
confluent-kafka  using the command in the "Install from
source from PyPi" section below.Install AvroProducer and AvroConsumer$ pip install "confluent-kafka[avro]"Install from source from PyPi(requires librdkafka + dependencies to be installed separately):$ pip install --no-binary :all: confluent-kafkaFor source install, seePrerequisitesbelow.Broker CompatibilityThe Python client (as well as the underlying C library librdkafka) supports
all broker versions >= 0.8.
But due to the nature of the Kafka protocol in broker versions 0.8 and 0.9 it
is not safe for a client to assume what protocol version is actually supported
by the broker, thus you will need to hint the Python client what protocol
version it may use. This is done through two configuration settings:broker.version.fallback=YOUR_BROKER_VERSION(default 0.9.0.1)api.version.request=true|false(default true)When using a Kafka 0.10 broker or later you don't need to do anything
(api.version.request=trueis the default).
If you use Kafka broker 0.9 or 0.8 you must setapi.version.request=falseand setbroker.version.fallbackto your broker version,
e.gbroker.version.fallback=0.9.0.1.More info here:https://github.com/edenhill/librdkafka/wiki/Broker-version-compatibilitySSL certificatesIf you're connecting to a Kafka cluster through SSL you will need to configure
the client with'security.protocol': 'SSL'(or'SASL_SSL'if SASL
authentication is used).The client will use CA certificates to verify the broker's certificate.
The embedded OpenSSL library will look for CA certificates in/usr/lib/ssl/certs/or/usr/lib/ssl/cacert.pem. CA certificates are typically provided by the
Linux distribution'sca-certificatespackage which needs to be installed
throughapt,yum, et.al.If your system stores CA certificates in another location you will need to
configure the client with'ssl.ca.location': '/path/to/cacert.pem'.Alternatively, the CA certificates can be provided by thecertifiPython package. To use certifi, add animport certifiline and configure the
client's CA location with'ssl.ca.location': certifi.where().PrerequisitesPython >= 2.7 or Python 3.xlibrdkafka>= 1.6.0 (latest release is embedded in wheels)librdkafka is embedded in the macosx manylinux wheels, for other platforms, SASL Kerberos/GSSAPI support or
when a specific version of librdkafka is desired, following these guidelines:ForDebian/Ubuntubased systems, add this APT repo and then dosudo apt-get install librdkafka-dev python-dev:http://docs.confluent.io/current/installation.html#installation-aptForRedHatandRPM-based distros, add this YUM repo and then dosudo yum install librdkafka-devel python-devel:http://docs.confluent.io/current/installation.html#rpm-packages-via-yumOnOSX, usehomebrewand dobrew install librdkafkaLicenseApache License v2.0KAFKA is a registered trademark of The Apache Software Foundation and has been licensed for use
by confluent-kafka-python. confluent-kafka-python has no affiliation with and is not endorsed by
The Apache Software Foundation.Developer NotesInstructions on building and testing confluent-kafka-python can be foundhere.