Package: papfa
PapfaPapfa is a high level pub/sub pattern Python library.Quick Start for DjangoInstall Papfa:$ pip install papfaAdd Papfa to your settings:PAPFA={'BROKER':'KAFKA','KAFKA_BOOTSTRAP_SERVERS':...,'KAFKA_SASL_PASSWORD':...,'KAFKA_SASL_USERNAME':...,'KAFKA_SASL_MECHANISM':...,'KAFKA_SECURITY_PROTOCOL':...,'GROUP_ID_PREFIX':...,'SCHEMA_REGISTRY_URL':...,'SCHEMA_REGISTRY_BASIC_AUTH':...,'CONSUMER_MIDDLEWARES':[...]}ConsumerAdd <django_project>.papfa.pyfrompapfaimportPapfapapfa_app=Papfa()Import papfa in <django_project>.__init__.pyfrom<django_project>.papfaimportpapfa_app__all__=['papfa_app']Add consumer in<app>/consumers.py:frompapfa.consumersimportconsumerfrompapfa.dtosimportRecord@consumer(topic='topic_name')defmy_consumer(messages:List[Record]):print(messages)Use papfa cli to consumepapfaconsume-a<django_app><consumer_name>Example of project structure:└── shop
    ├── shop
    │   ├── __init__.py
    │   ├── papfa.py
    │   ├── settings.py
    │   ├── urls.py
    │   └── wsgi.py
    ├── app1
    │   ├── __init__.py
    │   ├── admin.py
    │   ├── apps.py
    │   ├── consumers.py
    │   ├── migrations
    │   │   └── __init__.py
    │   ├── models.py
    │   ├── tests.py
    │   └── views.py
    ├── app2
    │   ├── __init__.py
    │   ├── admin.py
    │   ├── apps.py
    │   ├── consumers.py
    │   ├── migrations
    │   │   └── __init__.py
    │   ├── models.py
    │   ├── tests.py
    │   └── views.py
    └── manage.pyProducerProduce Message:fromdataclassesimportdataclassfromdataclasses_avroschemaimportAvroModelfrompapfa.producerimportget_message_producerfrompapfa.dtosimportRecord@dataclassclassUser(AvroModel):name:strage:intr1=Record(value=User(name='thom',age=53))r1=Record(value=User(name='jonny',age=50))message_producer=get_message_producer(topic='topic_name',User)message_producer.produce(messages=[r1,r2])CLIPapfa provides a command line interface to consume and monitor consumers.CommandsCommandDescriptionlistlist of all consumersconsumeconsume messages from a known consumerstatsshow stats of a consumerMiddlewarePapfa provides middlewares for both consumers and producers. You can implement your own middleware by extending thepapfa.middlewares.consumer.ConsumerMiddlewareandpapfa.middlwares.producer.ProducerMiddlewareclass.Default Middlewarespapfa.middlewares.consumer.ConsumedMessageStatsMiddleware- Logs the last message consumed by each topic - consumer groupSerializationFor Now Papfa only support confluent avro serialization with schema registry.BrokerFor Now Papfa only support Apache Kafka.FeaturesBatch Processing (Commit per batch)Consumed Messages StatsTodosHandle IdempotencyAdd Other Brokers & SerializersHandle Multiple Broker ClusterCreditsThis package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template.History0.1.11 (2023-01-23)Fix meta_data bug0.1.10 (2023-01-23)Add Transactional ProducerAdd More Logs for ConsumersFix Avro Dependency0.1.9 (2023-01-03)Add consumer_kwargs in consumer0.1.8 (2022-12-07)Add kafka_config in consumer0.1.7 (2022-11-29)Add deserialize key flag0.1.6 (2022-11-27)Add Python 3.10 Support0.1.5 (2022-06-08)Fix AppRegistryNotReady error when importing celery0.1.4 (2022-06-01)Fix group id with prefix in consumer0.1.3 (2022-05-31)Fix auto commit in batch processing.0.1.1 (2022-05-28)Fix value error caused by schema registry invalid configuration.0.1.0 (2022-05-11)First release on PyPI.