Package: microservices-messaging-layer
TriskelionA Kafka Messaging helper for MicroservicesService descriptionTriskelion provides your code with a helper for Kafka and Confluent Kafka producers.
The end goal of this project is to become a messages communication layer for microservices.
However, The code is at its earlier stage and there is room for improvements. If you want to contribute to it please contact me at antonio.dimariano[AT]gmail.com
If you find a bug please submit ithereI would like to thank all theConfluent Community on Slackfor the priceless help during my learning path.Confluent Kafka Producer ConfigurationBrokersENV variables nameVALUEbrokersmybroker1:9093,mybroker2:9093,mybroker3:9093schema_registryhttp://my_avro_schema_registry:8081SASL IdentificationPlease refer tohttps://docs.confluent.io/3.0.0/kafka/sasl.html.
If your broker requires SASL authentication, these are the ENVironment variables to includeENV variables nameVALUEsasl_mechanismsPLAINsecurity_protocolSASL_SSLsasl_usernameYOUR USERNAME HEREsasl_passwordYOUR PASSWORD HEREIf you schema registry requires authentication, these are the ENVironment variables to includeENV variables nameVALUEschema_registry_basic_auth_user_infoauthentication stringschema_registry_basic_auth_credentials_sourceUSER_INFOSSL certificate for brokers and schema registryIf your brokers and schema registry servers require a SSL certificate, these are theENV variables nameVALUEsecurity_protocolssl stringYou have to place your certificate inHow to use itConfluent Kafka ProducerTheproduce_messagemethod accepts a list of messages. Messages are dispatched asynchronously to the Kafka Broker
The module provides you with two different Producers' class. Both of them have a public methodproducer_message.
However, If your are working with AVRO topics, you have to dofrom messaging_middleware.confluent_producer.AVROProducer import AVROPRODUCER.
Otherwise you canfrom messaging_middleware.confluent_producer.ConfluentProducer import ConfluentProducerThe following is an example how to produce a message in both cases.frommessaging_middleware.confluent_producer.ConfluentProducerimportConfluentProducerfrommessaging_middleware.confluent_producer.AVROProducerimportAVROPRODUCERimportosdefault_key_value={"service_name":"test"}classOutboundCommunicationLayer:def__init__(self,topic,brokers_uri,security_protocol='plaintext',schema_registry=None):print("OS:",os.environ.get('brokers'),os.environ.get('schema_registry'))ifschema_registryisNone:self.producer=ConfluentProducer(brokers_uri=brokers_uri,topic=topic,security_protocol=security_protocol)else:self.avro_producer=AVROPRODUCER(brokers_uri=brokers_uri,schema_registry_url=schema_registry,topic=topic,security_protocol=security_protocol)defproduce_topic(self,value):self.producer.produce_message(message=value)defproduce_avro_topic(self,value,key=default_key_value):self.avro_producer.produce_message(value=value,key=key)if__name__=="__main__":o=OutboundCommunicationLayer(topic='my-test-2',brokers_uri=os.environ.get('brokers'),schema_registry=os.environ.get('schema_registry'))o.produce_avro_topic(value={"my_text":"ciaociaociaociao"},key=default_key_value)Using Synchronous ProducerEven though it is not always the best practise and it has its issues (https://github.com/edenhill/librdkafka/wiki/FAQ#why-is-there-no-sync-produce-interface) , there is no harm to have a synchronous producer.
The module provides you with two classes toIntegrated Logging SystemThe  Logger class aims to be used to print log messages and/or to produce logging message to an AVRO topic.
It uses the AVRO_PRODUCER and by default it produces messages to the AVRO topictcservicesmonitorwith the following schema
If you want to configure the Confluent Kafka producer, you have to pass ainit_kafka=1when creating your instance.logger=Logger(init_kafka=1,brokers_uri=os.environ.get('brokers'),schema_registry_url=os.environ.get('schema_registry_url'))tcservicesmonitor-value{"service_name":"string","last_operation":"string","timestamp":"string","operation_result":"string","operation_description":"string","error_description":"string"}tcservicesmonitor-key"service_name":"string"}If you want to use another topic you have to specify it in the ENV variablemonitoring_topicIn order to use the Logging system, please refer to the configuration used to set up aConfluent Kafka Producerimportosfrommessaging_middleware.Logger.LoggerimportLoggerfromdatetimeimportdatetime,timezonelogger=Logger(init_kafka=1,brokers_uri=os.environ.get('brokers'),schema_registry_url=os.environ.get('schema_registry_url'))logger.print_log('info','My log ',10)logger.print_log('debug','my second log')message_to_produce={"service_name":"my-test-microservice","last_operation":"last-completed-operation","timestamp":datetime.now(timezone.utc).replace(microsecond=0).isoformat(),"operation_result":"SUCCESS","operation_description":"The operation description","error_description":""}logger.produce_msg(message_to_produce=message_to_produce,schema_key={"service_name":"my-test-service"})