Package: wunderkafka
WunderkafkaThe power of librdkafka forhumanspythonsWunderkafka provides a handful of facades for C-powered consumer/producer. It's built on top of theconfluent-kafkaFor a quick view on what is going on, please checkQuickstartandDocumentationInstallation process describedhereFeatures#TypeSafe librdkafka configInstead of passing just adictto consumer/producer config, thepydantic-poweredconfig is used.
It is extracted directly from librdkafka'sCONFIGURATION.mdwith some rough parsing.Confluent & Cloudera Schema Registry supportConfluent is used as-is, but hortonworks/cloudera schema registry client and (de)serialization protocol are implemented as well (no "admin" methods support).Building KitWunderkafka allows you to relatively simply build your own transport for message (de)serialization and eliminate boilerplates for typical cases.Pre-defined config modelsimportosfromfunctoolsimportpartialfrompydanticimportfield_validator,Fieldfromwunderkafka.timeimportnowfromwunderkafkaimportSRConfig,ConsumerConfig,SecurityProtocol,AvroConsumer# If you are a fan of 12 factors, you may want to config via env variablesclassOverridenSRConfig(SRConfig):url:str=Field(alias='SCHEMA_REGISTRY_URL')@field_validator('sasl_username')@classmethoddeffrom_env(cls,v)->str:# And to use 'native' kerberos envsreturn'{0}@{1}'.format(os.environ.get('KRB5_USER'),os.environ.get('KRB5_REALM'))# Or you want to override some defaults by default (pun intended)classOverridenConfig(ConsumerConfig):# Consumer which do not commit messages automaticallyenable_auto_commit:bool=False# And knows nothing after restart due to new gid.group_id:str='wunderkafka-{0}'.format(now())# More 12 factorsbootstrap_servers:str=Field(env='BOOTSTRAP_SERVER')security_protocol:SecurityProtocol=SecurityProtocol.sasl_sslsasl_kerberos_kinit_cmd:str=''sr:SRConfig=OverridenSRConfig()@field_validator('sasl_kerberos_kinit_cmd')@classmethoddefformat_keytab(cls,v)->str:ifnotv:return'kinit{0}@{1}-k -t{0}.keytab'.format(os.environ.get('KRB5_USER'),os.environ.get('KRB5_REALM'))# Still allowing to set it manuallyreturnstr(v)# After this, you can `partial` your own Producer/Consumer, something like...MyConsumer=partial(AvroConsumer,config=OverridenConfig())# ORclassMyConsumer(AvroConsumer):def__init__(self,config:ConsumerConfig=OverridenConfig()):super().__init__(config)Building your own transportfromtypingimportOptionalfrompydanticimportFieldfromwunderkafka.config.generatedimportenumsfromwunderkafka.consumers.bytesimportBytesConsumerfromwunderkafka.schema_registryimportClouderaSRClientfromwunderkafka.hotfixes.watchdogimportcheck_watchdogfromwunderkafka.serdes.headersimportConfluentClouderaHeadersHandlerfromwunderkafka.consumers.constructorimportHighLevelDeserializingConsumerfromwunderkafka.schema_registry.cacheimportSimpleCachefromwunderkafka.schema_registry.transportimportKerberizableHTTPClientfromwunderkafka.serdes.avro.deserializersimportFastAvroDeserializerfromwunderkafkaimportSRConfig,ConsumerConfig,SecurityProtocolclassSRConfig(SRConfig):url:str=Field(alias="SCHEMA_REGISTRY_URL")security_protocol:SecurityProtocol=SecurityProtocol.sasl_sslsasl_mechanism:str="SCRAM-SHA-512"sasl_username:str=Field(alias="SASL_USERNAME")sasl_password:str=Field(alias="SASL_PASSWORD")classOverridenConsumerConfig(ConsumerConfig):enable_auto_commit:bool=Falseauto_offset_reset:enums.AutoOffsetReset=enums.AutoOffsetReset.earliestbootstrap_servers:str=Field(env="BOOTSTRAP_SERVERS")security_protocol:SecurityProtocol=SecurityProtocol.sasl_sslsasl_mechanism:str="SCRAM-SHA-512"sasl_username:str=Field(alias="SASL_USERNAME")sasl_password:str=Field(alias="SASL_PASSWORD")sr:SRConfig=Field(default_factory=SRConfig)# Pydantic/FastAPI style, but you can inherit from `HighLevelDeserializingConsumer` directlydefMyAvroConsumer(config:Optional[ConsumerConfig]=None,)->HighLevelDeserializingConsumer:config=configorOverridenConsumerConfig()config,watchdog=check_watchdog(config)returnHighLevelDeserializingConsumer(consumer=BytesConsumer(config,watchdog),schema_registry=ClouderaSRClient(KerberizableHTTPClient(config.sr),SimpleCache()),headers_handler=ConfluentClouderaHeadersHandler().parse,deserializer=FastAvroDeserializer(),)Avro on-the-fly schema derivationSupportsdataclassesandpydantic.BaseModelfor avro serialization powered bydataclasses-avroschemaand some rough "metaprogramming":# dataclass to AVRO schema examplefromdataclassesimportdataclassfromdataclasses_avroschemaimportAvroModel@dataclassclassSomeData(AvroModel):field1:intfield2:strfor a topictopic_namewill become{"type":"record","name":"topic_name_value","fields":[{"name":"field1","type":"long"},{"name":"field2","type":"string"}]}and# pydantic.BaseModel to AVRO schema examplefromtypingimportOptionalfrompydanticimportBaseModelclassEvent(BaseModel):id:Optional[int]ts:Optional[int]=NoneclassMeta:namespace="any.data"for a topictopic_namewill become{"type":"record","name":"topic_name_value","namespace":"any.data","fields":[{"type":["long","null"],"name":"id"},{"type":["null","long"],"name":"ts","default":null}]}