Package: nameko-kafka
Nameko-KafkaKafka extension forNamekomicroservice framework.IntroductionThis is a Nameko microservice frameworkextensionto support
Kafka entrypoint and dependency. The motivation behind the project is issue569.Nameko-kafkaprovide a simple implementation of the entrypoint based on the approach bycalumpeterwebb.
It also includes a dependency provider for publishing Kafka messages from within a Nameko service.InstallationThe package is supports Python >= 3.5$pipinstallnameko-kafkaUsageThe extension can be used for both, a service dependency and entrypoint. Example usage for both cases are shown in the
following sections.DependencyThis is basically apython-kafkaproducer in the form of Nameko dependency.
Nameko uses dependency injection to instantiate the producer. You just need to declare it in your service class as shown:fromnameko.rpcimportrpcfromnameko_kafkaimportKafkaProducerclassMyService:"""My microservice"""name="my-service"# Kafak dependencyproducer=KafkaProducer(bootstrap_servers='localhost:1234')@rpcdefmethod(self):# Publish message using dependencyself.producer.send("kafka-topic",value=b"my-message",key=b"my-key")HereKafkaProduceraccepts all options valid forpython-kafka'sKafkaProducer.EntrypointYou can use thenameko_kafka.consumedecorator in your services to process Kafka messages:fromnameko_kafkaimportconsumeclassMyService:"""My microservice"""name="my-service"@consume("kafka-topic",group_id="my-group",bootstrap_servers='localhost:1234')defmethod(self,message):# Your message handlerhandle_message(message)Theconsumedecorator accepts all the options valid forpython-kafka'sKafkaConsumer.On top of the defaultpython-kafka's autocommit feature, the entrypoint also comes with support for three different
types of offset commit strategies:at least once,at most onceandexactly once. The three strategies correspond
to the different message delivery semantics achievable in Kafka. Examples for each are shown in the following subsections.At Least Oncefromnameko_kafkaimportconsume,SemanticclassMyService:"""My microservice"""name="my-service"# At least once semantic consumer@consume("kafka-topic",group_id="my-group",bootstrap_servers='localhost:1234',semantic=Semantic.AT_LEAST_ONCE)defmethod(self,message):# Your message handlerhandle_message(message)At Most Oncefromnameko_kafkaimportconsume,SemanticclassMyService:"""My microservice"""name="my-service"# At most once semantic consumer@consume("kafka-topic",group_id="my-group",bootstrap_servers='localhost:1234',semantic=Semantic.AT_MOST_ONCE)defmethod(self,message):# Your message handlerhandle_message(message)Exactly OnceThe exactly once semantic requires a persistent storage to save message offsets. Such a persistent store can be
implemented using theOffsetStorageinterface provided by Nameko-kafka. There can be various backend implementations
like RDBMS, NoSQL databases, etc. Support for some comes out of the box:MongoDB Storagefromnameko_kafkaimportconsume,Semanticfromnameko_kafka.storageimportMongoStoragefrompymongoimportMongoClientclassMyService:"""My microservice"""name="my-service"# At most once semantic consumer@consume("kafka-topic",group_id="my-group",bootstrap_servers='localhost:1234',semantic=Semantic.EXACTLY_ONCE,storage=MongoStorage(# MongoDB backend clientclient=MongoClient('localhost',27017),# Database to use for storagedb_name="database-name",# Collection to use for storagecollection="collection-name"))defmethod(self,message):# Your message handlerhandle_message(message)Note: If thedb_nameandcollectionarguments are not specified, the default value of"nameko_kafka_offsets"and"offsets"will be used by the storage respectively.SQL StoragePart of v0.3.0S3 StoragePart of v0.4.0Azure Block StoragePart of v0.5.0Create Custom StorageYou can create your own offset storage by implementing theOffsetStorageinterface. It exposes the following methods:fromnameko_kafka.storage.baseimportOffsetStorageclassMyStorage(OffsetStorage):"""My custom offset storage."""defsetup(self):"""Method for setup of the storage."""defstop(self):"""Method to teardown the storage."""defread(self,topic,partition):"""Read last stored offset from storage forgiven topic and partition.:param topic: message topic:param partition: partition number of the topic:returns: last committed offset value"""defwrite(self,offsets):"""Write offsets to storage.:param offsets: mapping between topic-partitiontuples and corresponding latest offset value,e.g.{("topic-1", 0): 1,("topic-1", 1): 3,("topic-2", 1): 10,...}"""ConfigurationsThe extension configurations can be set in a namekoconfig.yamlfile, or
by environment variables.Config File# Config for entrypointKAFKA_CONSUMER:bootstrap_servers:'localhost:1234'retry_backoff_ms:100...# Config for dependencyKAFKA_PRODUCER:bootstrap_servers:'localhost:1234'retries:3...Environment Variables# Config for entrypoint
KAFKA_CONSUMER='{"bootstrap_servers": "localhost:1234", "retry_backoff_ms": 100}'

# Config for dependency
KAFKA_PRODUCER='{"bootstrap_servers": "localhost:1234", "retries": 3}'MilestonesKafka EntrypointKafka DependencyCommit strategies:ALMOST_ONCE_DELIVERYAT_LEAST_ONCE_DELIVERYEXACTLY_ONCE_DELIVERYCommit storage forEXACT_ONCE_DELIVERYstrategyDevelopersFor development a kafka broker is required. You can spawn one using thedocker-compose.ymlfile in thetestsfolder:$cdtests
$docker-composeup-dTo install all package dependencies:$pipinstall-r.[dev]or
$makedepsOther useful commands:$pytest--cov=nameko_kafkatests/# to get coverage reportor
$makecoverage

$pylintnameko_kafka# to check code quality with PyLintor
$makelintContributionsIssue reports and Pull requests are always welcomed. Thanks!