Package: citizenk
CitizenKCitizienKis a simple but powerful Python Library for developing reactive async Kafka microservices, built on top ofConfluent Kafka Python,FastAPIandPydantic.CitizenK Replicatoris an additional tool that we developed using the same technology to simplify data transfer between production and staging environments. It's not a substitution for Confluent's replicator which is a much more robust tool for replicating data between multiple production environmentsHow we got here...We exclusively use Python for service development. Our work involves crafting web services, creating ETL code, and engaging in data science, all within the Python ecosystem. A few years back, as we embarked on the Lanternn project, our quest led us to seek a Python library that could facilitate the construction of distributed, scalable processing pipelines built on top of Kafka. These pipelines needed to accommodate both stateless and stateful microservices seamlessly. After extensive exploration, we found that the most suitable solution at the time was Faust.Faust is a stream processing library that borrows concepts from Kafka Streams and brings them into the Python realm. Beyond this, Faust boasts an impressive array of features, including a robust web server, comprehensive schema validation and management, all built upon an agents/actors architecture. With such a compelling set of attributes, it was impossible for us to resist its allure.We went on to develop numerous services utilizing Faust, and by and large, we were quite content with the results. However, as time progressed, we came to realize that Kafka Streams was not the ideal fit for our needs. Its complexity made it challenging to manage, and we found simpler alternatives for state management, such as Redis, to be more suitable. Additionally, concerns began to emerge about the long-term viability of Faust, particularly in the absence of its creator Ask Solem. Moreover, the underlying Kafka libraries it relied upon, aiokafka and python-kafka, lacked the robust community support necessary to address the stability issues we encountered.Concurrently, we observed that frameworks like FastAPI and Confluent Kafka, which we were already using, enjoyed strong backing from vibrant and sizable communities. This realization led us to explore the possibility of combining these frameworks to establish a new foundation for our pipelines, one that would offer greater stability and long-term viability, and would be easy to migrate to from Faust.The choice of the name "CitizenK" embodies our belief that Python should occupy a prominent position within the Kafka ecosystem. It also draws inspiration from Kafka's renowned novel, "The Trial," which chronicles the plight of Josef K., a man ensnared and prosecuted by an enigmatic, distant authority. The nature of his alleged transgression remains shrouded in mystery, a narrative that resonates with our journey in the world of Kafka.Existing toolsFaustFastkafkaTutorialYou can see an example of how to use CitizenK in the demo appCreating a CitizienK appFirst, we create a CitizenK app similar to how we create a FastAPI app, but with additional arguments:kafka_config: provides configuration for connecting and configuring the Kafka clientapp_name: Mainly used as the consumer group nameapp_type: SINK (consumer only), SOURCE(producer only) or TRANSFORM(producer-consumer)auto_generate_apis: Will auto generate FastAPI to consume and produce to workers and topicsagents_in_thread: Will run the consumer agents in a thread and not in an async loopconsumer_group_init_offset: Where to start consuming when the consumer group is createdconsumer_group_auto_commit: commit after consume / commit after processing completed successfully in all agentsexit_on_agent_exception: exit the service if there is exception in an agentapp=CitizenK(kafka_config=config.source_kafka,app_name="citizenk",app_type=AppType.TRANSFORM,debug=True,title="CitizenK Demo App",auto_generate_apis=True,agents_in_thread=config.AGENTS_IN_THREAD,api_router_prefix=prefix,api_port=config.API_PORT,schema_registry_url=config.KAFKA_SCHEMA_REGISTRY,version=config.VERSION,consumer_group_init_offset="latest",consumer_group_auto_commit=True,consumer_extra_config=config.KAFKA_CONSUMER_EXTRA_CONFIG,producer_extra_config=config.KAFKA_PRODUCER_EXTRA_CONFIG,exit_on_agent_exception=True,openapi_url=prefix+"/openapi.json",docs_url=prefix+"/docs",license_info={"name":"Apache 2.0","url":"https://www.apache.org/licenses/LICENSE-2.0.html",},)Creating CitizienK topicsNext, we create topics for the app and define the model for the topics using PydanticTopics can be either INPUT, OUTPUT or BIDIRclassVideo(JSONSchema):camera_id:intpath:strtimestamp:datetimeclassProcessedVideo(JSONSchema):camera_id:intpath:strtimestamp:datetimevalid:boolt1=app.topic(name="B",value_type=Video,topic_dir=TopicDir.BIDIR)t2=app.topic(name="C",value_type=ProcessedVideo,topic_dir=TopicDir.BIDIR)t3=app.topic(name="D",value_type=ProcessedVideo,topic_dir=TopicDir.OUTPUT)Schemas can also be AVROclassAvroProcessedVideo(AvroBase):camera_id:intpath:strtimestamp:datetimevalid:boolt4=app.topic(name="E",value_type=AvroProcessedVideo,topic_dir=TopicDir.BIDIR,schema_type=SchemaType.AVRO,)Creating CitizienK agentsAnd lastly, we create gents that process the Kafka messages.Agents can listen to multiple topics and accept either values or the entire Kafka event (key, value, offset, partition, timestamp...). Agents can also accept a self argument to get a reference to the Agent object.In non auto commit apps, offsets are committed only after all agents processed the event successfully.topics: one or more topics to processbatch_size: specify desired batch_size. Default = 1batch_timeout: How long to wait for batch to arrive: Default = 10 seconds@app.agent(topics=t1,batch_size=100)asyncdefprocess_videos_t1(events:list[KafkaEvent]):# Process incoming videoforeventinevents:camera_id=event.value.camera_idvideo_counts[camera_id]+=1v=ProcessedVideo(camera_id=camera_id,path=event.value.path,timestamp=event.value.timestamp,valid=bool(camera_id%2),)t2.send(value=v,key=str(v.camera_id))@app.agent(topics=t2,batch_size=100)asyncdefprocess_videos_t2(values:list[BaseModel]):# Process incoming videoforvalueinvalues:ifvalue.valid:t3.send(value=value,key=str(value.camera_id))Auto endpointsTo help debug and evaluate the service, CitizenK automatically creates web endpoints that help you send messages to topics and agents.info: get service infotopics: send events to topicsagents: send events directly to agents, bypassing topicsstats: get Kafka stats for producer and consumerCreating additional CitizienK endpointsJust like any other FastAPI app, you can create get, post and put endpoints that either interact with Kafka or perform some other tasks, non Kafka related@router.post("/events",response_class=JSONResponse)asyncdefproduce_video_events(values:list[Video],topic:str=Query(),):"""Sends events to the given topic"""iftopicnotinapp.topics:raiseHTTPException(status_code=400,detail="Topic not supported by app")t=app.topics[topic]forvinvalues:t.send(value=v,key=str(v.camera_id))return{"status":"ok"}@router.get("/topics",response_class=JSONResponse)asyncdefget_source_topics():"""Returns the list of topics from the source kafka"""admin=KafkaAdapter(config.source_kafka)topics=sorted(list(admin.get_all_broker_topics().keys()))return{"topics":topics}Multiple workers behind a load balancerCitizenK includes two special decorators for scenarios where the service has multiple workers behind a load balancer and the web request needs to reach a specific worker that holds a partition.topic_router: forwards the request based on the topic and key (JSON / HTML)broadcast_router: aggregates the responses from all workers into a single JSONBoth routers support GET, POST, PUT and DELETE commands@router.get("/topic_test",response_class=JSONResponse)@app.topic_router(topic=t1,match_info="camera_id")asyncdeftest_topic_router(request:Request,camera_id:int):"""Returns the list of groups from the target kafka"""return{"key":camera_id,"count":video_counts[camera_id]}@router.get("/broadcast_test",response_class=JSONResponse)@app.broadcast_router()asyncdeftest_broadcast_router(request:Request):"""Returns the list of groups from the target kafka"""returnvideo_countsWebsocketCitizenK support for Websocket agents@app.agent(topics=t2,batch_size=100,websocket_route=prefix+"/ws")asyncdefwebsocket_agent(values:list[BaseModel])->str:values=[json.loads(v.json())forvinvaluesifnotv.valid]returnjson.dumps(values,indent=4)This agent exposes a WebSocket endpoint for one or more clients to connect to. It then processes incoming Kafka messages from topic t2 and sends the returned string value to all the existing live WebSocket "/ws" connections. The main use case for this is to bridge between Kafka and Websocket. One possible use case for this feature is to send filtered Kafka events to a web app or mobile app.The other direction frontend --> Kafka is probably easier to implement with a normal REST post endpoint and is not supported yet.Things to be aware of...CitizenK is a single-threaded async app. i.e. If a coroutine spends too much time in processing without awaiting IO, it will block other coroutines from running. Specifically, when using a load balancer with health checks, it's important to pay attention to the time between health checks and see that it's higher than the longest-running agent. Fixed using:agents_in_threadTo help tune the service. CitizenK includes the concept of batch size:i.e. how many events to consume and process every batch across all agents.Additionally like any other Kafka service. it's important to tune several kafkaconsumerandproducerconfigs. Specifically ensure rebalancing is not triggered unintentionally: Alternatively this list includesall Kafka configsConsumer:fetch.max.bytes (50 Mbytes): The maximum amount of data the server should return for a fetch request. Reduce if processing each record takes significant time.max.poll.records(500): The maximum number of records returned in a single call to poll().max.poll.interval.ms (5 min): The maximum delay between invocations of poll() when using consumer group managementGroup rebalancing in stateful services:Prefer static membership + Increase session.timeout.ms to 2-5 minutes (how long it takes a new service to come up)partition.assignment.strategy': 'range'Producer:linger.ms(0): Important to set to 0/5/10/50/200 on moderate/high loadbatch.size(16K): Increase if sending large buffers to KafkaBoth:compression.type(none): gzip, snappy, or lz4More explanation in here:Solving My Weird Kafka Rebalancing Problems & Explaining What Is Happening and Why?CitizenK ReplicatorScenariosStaging environmentI have a staging environment and I want to replicate some production topics to itAt some point I want to produce the staging topics in the staging environment using a staging service. So I switch off the replication and populate the same staging topic with real data.When I finish the testing in staging, I want to switch back to production, so that I can save on costs.If the workload is high, I want to replicate most (i.e. 90%) of the messages from production and only produce just a little (i.e. 10%) of the data from staging. This way in the same topic, I will have mixed data and potential schema from the two environmentsWhen switching between environments (i.e. on configuration change), I want to change the offset to the latest on the new topic, so that the handover is not too chaoticI also want to delete the consumer group of the service in staging, so that when it come back up again, it won't see a lag.Additionally sometimes, I want to migrate data between production and staging due to schema change or different identities.Dev environment + live dataWhen I test a service locally or in a dev environment, possibly with a local Kafka, I want the local Kafka to have real data, so that I can test the service for a long period of time with live data.Theoretically, I can connect the dev service to the staging or production Kafka cluster, however, this presents a stability/security risk to the remote cluster. There is also a risk that the service will join a consumer group and participate accidentally in the remote workload. This approach also prevents parallel testing as there can be a conflict between the consumers.So one solution would be to replicate the topics from staging to the local/dev Kafka, maybe with some filtering to reduce the load, so that the local service is not overwhelmed with too much dataOn premise kafka -- cloud kafka bridgeI have a local Kafka and I want to replicate some topics to the remote cloudYou can use this tool for this scenario, however Confluent replicator or Kafka MirrorMaker are probably more suitableDev environment + replayed dataWhen I test a service locally or in a dev environment, possibly with a local Kafka, I want the local Kafka to replay historical/simulated messages from a file.this scenario is a bit different from the previous ones, as there is no Kafka consumer, just a producer. And you can say that it is more of a tool than a service.The messages are read from a file with a timestamp (one file for each topic), and injected into the right topic with the correct timing keeping the same gap between now, and the initial timestamp.Cluster -- Cluster replicationYou can use this tool for this scenario, however Confluent replicator or Kafka MirrorMaker are probably more suitableExisting toolsConfluent replicator: Looks like a good tool, but not open source, expensiveKafka Mirror Maker: Open source but doesn't support filteringkcat: Nice tool, but not for these scenariosImplementation detailsUsing containerised pythonBased on Confluent Kafka API + FastAPDoes not create the topics or partitions automatically. It assumes they exists and configuredDeployed as a distributed serviceFilter based on JMESPath for JSON messagesAllow two consumer options: with consumer group, without consumer groupwrite code following DDD principlesConfigurationLOG_LEVEL: service log levelJSON_LOGGING: Use json loggingAPI_PREFIX: API prefixFILE_DATA_PATH: Location of json files when reading and writing topics from fileKAFKA_SOURCE_SERVER_URL: Source Bootstrap ServersKAFKA_SOURCE_USE_TLS: Enable Source SSL: 0,1KAFKA_SOURCE_SASL_MECHANISM: Source SASL mechanism: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512KAFKA_SOURCE_SASL_USERNAME: Source SASL usernameKAFKA_SOURCE_SASL_PASSWORD: Source SASL passwordKAFKA_SOURCE_GROUP_NAME: Source group name, or leave empty to consume without a consumer groupKAFKA_SOURCE_EXTRA_CONFIG_<KAFKA_CONFIG_KEY>: Any valid kafka consumer config (uppercase, replace . with _)KAFKA_TARGET_SERVER_URL: Target Bootstrap ServersKAFKA_TARGET_USE_TLS: Enable Target SSLKAFKA_TARGET_SASL_MECHANISM: Target SASL mechanism: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512KAFKA_TARGET_SASL_USERNAME: Target SASL usernameKAFKA_TARGET_SASL_PASSWORD: Target SASL passwordKAFKA_TARGET_EXTRA_CONFIG_<KAFKA_CONFIG_KEY>: Any valid kafka producer config (uppercase, replace . with _)READ_MAPPINGS_EVERY_SECONDS: How often to check for new mappings in the file systemCACULATE_STATS_EVERY_SECONDS: How often to calculate statsDELETE_GROUPS_EVERY_SECONDS: How often to check for new group deletionCurrent solution limitationsCurrently only supports JSON schema.APIAPI DescriptionUser InterfaceThe user interface allows you to add a new mapping and edit/delete an existing mapping.UsageProvide a JSON list of topic mappings in this format:[{"name":"File A to B","source_topic_name":"A","target_topic_name":"B","source_is_file":true},{"name":"Topic B to C","source_topic_name":"B","target_topic_name":"C"},{"name":"Topic C to D Using filter","source_topic_name":"C","target_topic_name":"D","valid_jmespath":"key == 'hello' && value.msg == 'world'","enabled":true},{"name":"TopicCtoD","source_topic_name":"C","target_topic_name":"D","enabled":false,"target_service_consumer_group":"service"},{"name":"Topic D to File E","source_topic_name":"D","target_topic_name":"E","target_is_file":true},{"name":"Disabled Topic U to File V","source_topic_name":"U","target_topic_name":"V","target_is_file":true,"enabled":false}]name: The mapping nameenabled: Enable / disable mappingsource_topic_name: The topic to read from in the source clustertarget_topic_name: The topic to write to in the target clustervalid_jmespath: filter criteriasource_is_file: If the source is a json filetarget_is_file: If the target is a json filetarget_service_consumer_group: The service consumer group to delete when replication is enabledTopic Level MappingTopic level mappings allows mapping of key/values when replicating a topic. This might be useful if for example the schema / enums / keys are different between the environmentsTo support this, the replicator supports value mappings for each topic that it consumes from the source in the following JSON format:There are two mapping formats:value.payload.product_id : map source product_id to target product_idkey:partition: map source key to target partition (drop entire msg if partition equals -1000){"key":{"1":10,"2":12},"value.payload.product_id":{"1001":1,"1002":12,"1003":14},"value.payload.user_name":{"A":"A name","B":"B name","C":"C name"},"key:partition":{"1":0,"2":0,"3":-1000,"4":1,"5":1,"6":1,}}StatsReturns a list of JSON stats for each mapping in the following format:{"time":"2023-05-25 18:20:43.875557","started":"2023-05-25 08:08:35.728313","queue":180,"mappings":[{"name":"Topic B to C","source_topic_name":"B","target_topic_name":"C","valid_jmespath":null,"target_service_consumer_group":null,"consumer_group_up":false,"assignments":[0,1,2],"lag":1258,"source_count":27739,"target_count":27739}]}Grafana IntegrationTo view the stats in Grafana, use the Infinity data source with the following settings:Consumer APITo simplify debug and to support other use cases, the replicator also includes an end point to consume messages from a given topic.LicenseApache License v2.0