Package: kafkaesk
kafkaeskTable Of ContentsAbout the ProjectPublishSubscribeAvoiding global objectManual commitkafkaesk contractWorkerDevelopmentExtensionsNamingAbout The ProjectThis project is meant to help facilitate effortless publishing and subscribing to events with Python and Kafka.Guiding principalHTTPLanguage agnosticContracts built on top ofKafkaAlternativesaiokafka: can be complex to scale correctlyguillotina_kafka: complex, tied toGuillotinafaust: requires additional data layers, not language agnosticconfluent kafka + avro: close but ends up being like grpc. compilation for languages. No asyncio.Consider this Python project as syntactic sugar around these ideas.PublishUsingpydanticbut can be done with pure JSON.importkafkaeskfrompydanticimportBaseModelapp=kafkaesk.Application()@app.schema("Content",version=1,retention=24*60*60)classContentMessage(BaseModel):foo:strasyncdeffoobar():# ...# doing something in an async funcawaitapp.publish("content.edited.Resource",data=ContentMessage(foo="bar"))A convenience method is available in thesubscriberdependency instance, this allow to header
propagation from the consumed message.importkafkaeskfrompydanticimportBaseModelapp=kafkaesk.Application()@app.schema("Content",version=1,retention=24*60*60)classContentMessage(BaseModel):foo:str@app.subscribe("content.*","group_id")asyncdefget_messages(data:ContentMessage,subscriber):print(f"{data.foo}")# This will propagate `data` record headersawaitsubscriber.publish("content.edited.Resource",data=ContentMessage(foo="bar"))SubscribeimportkafkaeskfrompydanticimportBaseModelapp=kafkaesk.Application()@app.schema("Content",version=1,retention=24*60*60)classContentMessage(BaseModel):foo:str@app.subscribe("content.*","group_id")asyncdefget_messages(data:ContentMessage):print(f"{data.foo}")Avoiding global objectIf you do not want to have global application configuration, you can lazily configure
the application and register schemas/subscribers separately.importkafkaeskfrompydanticimportBaseModelrouter=kafkaesk.Router()@router.schema("Content",version=1,retention=24*60*60)classContentMessage(BaseModel):foo:str@router.subscribe("content.*","group_id")asyncdefget_messages(data:ContentMessage):print(f"{data.foo}")if__name__=="__main__":app=kafkaesk.Application()app.mount(router)kafkaesk.run(app)Optional consumer injected parameters:schema: strrecord: aiokafka.structs.ConsumerRecordapp: kafkaesk.app.Applicationsubscriber: kafkaesk.app.BatchConsumerDepending on the type annotation for the first parameter, you will get different data injected:async def get_messages(data: ContentMessage): parses pydantic schemaasync def get_messages(data: bytes): give raw byte dataasync def get_messages(record: aiokafka.structs.ConsumerRecord): give kafka record objectasync def get_messages(data): raw json data in messageManual commitTo accomplish a manual commit strategy yourself:app=kafkaesk.Application(auto_commit=False)@app.subscribe("content.*","group_id")asyncdefget_messages(data:ContentMessage,subscriber):print(f"{data.foo}")awaitsubscriber.consumer.commit()SSLAdd these values to yourkafka_settings:ssl_context- this should be a placeholder as the SSL Context is generally created within the applicationsecurity_protocol- one of SSL or PLAINTEXTsasl_mechanism- one of PLAIN, GSSAPI, SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARERsasl_plain_username.sasl_plain_password.kafkaesk contractThis is a library around using kafka.
Kafka itself does not enforce these concepts.Every message must provide a json schemaMessages produced will be validated against json schemaEach topic will have only one schemaA single schema can be used for multiple topicsConsumed message schema validation is up to the consumerMessages will be consumed at least once. Considering this, your handling should be idempotentMessage format{"schema":"schema_name:1","data":{...}}Workerkafkaeskmymodule:app--kafka-servers=localhost:9092Options:--kafka-servers: comma separated list of kafka servers--kafka-settings: json encoded options to be passed tohttps://aiokafka.readthedocs.io/en/stable/api.html#aiokafkaconsumer-class--topic-prefix: prefix to use for topics--replication-factor: what replication factor topics should be created with. Defaults to min(number of servers, 3).Application.publishstream_id: str: name of stream to send data todata: class that inherits from pydantic.BaseModelkey: Optional[bytes]: key for message if it needs oneApplication.subscribestream_id: str: fnmatch pattern of streams to subscribe togroup: Optional[str]: consumer group id to use. Will use name of function if not providedApplication.schemaid: str: id of the schema to storeversion: Optional[int]: version of schema to storestreams: Optional[List[str]]: if streams are known ahead of time, you can pre-create them before you push dataretention: Optional[int]: retention policy in secondsApplication.configurekafka_servers: Optional[List[str]]: kafka servers to connect totopic_prefix: Optional[str]: topic name prefix to subscribe tokafka_settings: Optional[Dict[str, Any]]: additional aiokafka settings to pass inreplication_factor: Optional[int]: what replication factor topics should be created with. Defaults to min(number of servers, 3).kafka_api_version: str: defaultautoauto_commit: bool: defaultTrueauto_commit_interval_ms: int: default5000DevelopmentRequirementsDockerPoetrypoetryinstallRun tests:docker-composeupKAFKA=localhost:9092poetryrunpytesttestsExtensionsLoggingThis extension includes classes to extend Python's logging framework to publish structured log messages to a Kafka topic.
This extension is made up of three main components: an extendedlogging.LogRecordand some customlogging.Handlers.Seelogger.pyin examples directory.Log Recordkafkaesk.ext.logging.record.factoryis a function that will returnkafkaesk.ext.logging.record.PydanticLogRecordobjects.
Thefactory()function scans through anyargspassed to a logger and checks each item to determine if it is a subclass ofpydantid.BaseModel.If it is a base model instance andmodel._is_log_modelevaluates toTruethe model will be removed fromargsand added torecord._pydantic_data.
After thatfactory()will use logging's existing logic to finish creating the log record.HandlerThis extensions ships with two handlers capable of handlingkafkaesk.ext.logging.handler.PydanticLogModelclasses:kafakesk.ext.logging.handler.PydanticStreamHandlerandkafkaesk.ext.logging.handler.PydanticKafkaeskHandler.The stream handler is a very small wrapper aroundlogging.StreamHandler, the signature is the same, the only difference is that the handler will attempt to convert any pydantic models it receives to a human readable log message.The kafkaesk handler has a few more bits going on in the background.The handler has two required inputs, akafkaesk.app.Applicationinstance and a stream name.Once initialized any logs emitted by the handler will be saved into an internal queue.
There is a worker task that handles pulling logs from the queue and writing those logs to the specified topic.NamingIt's hard and "kafka" is already a fun name.
Hopefully this library isn't literally "kafkaesque" for you.