Package: pika-pydantic
Pika-PydanticAn opinionated Python implementation of the Producer-Consumer Pattern using RabbitMQ on top ofpikaandpydantic.IntroductionThispika_pydanticlibrary is a thin wrapper on top of thepikaandpydanticlibraries that makes it quick and easy to createProducer-Consumerworkers that interface with aRabbitMQmessage queue. For more information of why this library was created, see the Backstory section below in the documentation.I was inspired in many ways by what Sebastian created withFastAPIby building on top of good existing libraries.pika_pydanticattempts to follow that method in its own much simpler way for theasynchronous Producer-Consumer pattern using RabbitMQ.If you are creating a long chain of Producers and Consumers then thepika_pydanticlibrary can save quite a lot of boilerplate code and potential errors.InstallationTo install thepika_pydanticpackage using pippip install pika-pydanticOr using poetrypoetry add pika-pydanticDependencies - requires RabbitMQIn addition you need to have aRabbitMQinstance up and running that can receive and route the messages.If you have some familiarity with Docker, the easiest method is to spin up a docker container running RabbitMQ and use that as your message service. Thedocker-compose-rabbit.ymlprovides a simpledocker-composeconfiguration script for this.Alternatively, you can install RabbitMQ natively on your development machine or server, or link to a hosted RabbitMQ instance. More details on RabbitMQ installation can be found onthe official RabbitMQ documentationQuickstartThis simple example creates a simple message Producer-Consumer that passes around a message object.Create the pika connectionFirst we create a pika connection to the RabbitMQ systemimportpikaparameters=pika.URLParameters("amqp://guest:guest@localhost:5672/")connection=pika.BlockingConnection(parameters)This creates a normalpikablocking connection.Thepikadocumentation can be found herePika-pydantic Channel vs pika ChannelNow we deviate from the standardpikamethod. Instead of usingconnection.channel()or similar to create apika.BlockingChannelwe use thepika_pydantic.BlockingChannelobject instead. This object also initialises queues and adds various other useful methods on top of the standardpika.BlockingChannelobject.But before we do that we need to define the data validation and the queues that will constrain and validate our Producers and Consumers.Defining data modelsWe want to pass around a message data object that has a title, and text.
This data model is defined using thepika_pydantic.BaseModelwhich is a wrapper around the standardpydantic.BaseModelimportpika_pydanticclassMyMessage(pika_pydantic.BaseModel):"""A message"""title:strtext:strpika_pydantic.BaseModelobjects are pydantic.BaseModel objects with some additional elements for encoding and decoding the objects for RabbitMQ. See thepydantic documentationfor more details.Defining queuesWe also define the single message queue we will use in this example by definding anpika_pydantic.Queuesenum. The name on the left defines the Enum but also the RabbitMQ queue name. The value on the right defines the data model to use for validation.classMyQueues(pika_pydantic.Queues):MESSAGE=MyMessageThis object is the master that defines the valid queues and the corresponding data that all Producers and Consumers must use. Add more elements to this enum as you add queues and data models.pika_pydantic.Queuesobjects are a Pythonenum.Enumclass. The RabbitMQ queue name will be set to the same as the enum name (on the left), and the value on the right is thepika_pydantic.BaseModeldata model object that all Producers and Consumers on this queue need to use.Initialise the ChannelNow we can initialise the channel and we pass it thepika.connectionand thepika_pydantic.Queuesenum we just defined.channel=pika_pydantic.BlockingChannel(connection=connection,queues=MyQueues)pika_pydantic.BlockingChannelis apika.BlockingChannelobject with some additional methods attached that allow simpler creation of Consumers (listen()) and Producers (send())This object declares all the queues, and validates the message data on each queue and does the necessary encoding and decoding of the data for Consumers and Producers.Create a ConsumerTo create a new Consumer for this message queue we use the newchannel.listen(queue, callback)method. This validates the inputs and does the decoding needed for that particular queue. We define a callback as in pika and add the consumer to the channel.defcallback(channel,method,frame,data:MyMessage):print(f"Received message with title ({data.title}) and text ({data.text}).")channel.listen(queue=MyQueues.MESSAGE,callback=callback,auto_ack=True)Create a ProducerTo create a Producer we use the newchannel.send(queue, data)method. This takes the data object and does all the validation and encoding needed to pass it to the RabbitMQ queue.message=MyMessage(title="Important",text="Remember to feed the dog")channel.send(queue=MyQueues.MESSAGE,data=message)Start it runningAs with standard pika, the channel can start polling so that the defined Consumers start listening for messages on their queue.channel.start_consuming()Or to not block the thread and process the messages currently in the queue we can useconnection.process_data_events(time_limit=None)Other examplesTheexamplesfolder provides further examples and a suggested project folder structure that reuses thepika_pydanticelements across multiple Consumers and Producers.The backstoryAsynchronous messagingGood code structure generally separates concerns (jobs) between different modules. Microservices takes this one step further and separates jobs into different deployable systems that interact with each other.These different systems are interfaced through various APIs, usually called from one system to another in realtime.But some jobs are long lasting or resource hungry and this is where we can use asynchronous interfaces between the different systems.There is a lot of interest currently in Kafka as a system for managing these asynchronous jobs. But for most projects a simpler message queue such as RabbitMQ will do the job. It provides a way to pass data and a job onto another system, and that other system will pick up the job when it has resources to do so.The Producer-Consumer patternFor many purposes a system does some work and prepares some data. It then passes this on as a job for the next system element to work on when it has resources available. This is theProducer-Consumer Pattern.In a bit more detailA Producer completes some job, often resulting in some data artifact to be passed to the next stage.The Producer publishes this data to a message queueThe next job is a Consumer of this message queue. When it has resources available, it picks up the message and the published data and then does it work.This Consumer may itself also be a Producer publishing it's data to a different message queue for the next Consumer in the chain to take forward.RabbitMQ and the PythonpikalibraryIn the Python world there are good libraries for this, most notably is thepikalibrary that interfaces with a RabbitMQ message queue.pikais relatively simple and very flexible.But for my needs I wanted to use stricter software development principles and the flexibility ofpikatoo flexible. Specifically:My system has many Consumers and many Publishers. I wanted to be able to define thepikaboilerplate code to set up the connection, the queues and the channel in one central place for all the different jobs.I also wanted to restrict my Consumers and Publishers to only valid queues, so to do this I wanted to define the valid queues in an Enum to reduce strange bugs.I wanted to ensure that each Producer sending data sends the data in the right format and each Consumer picks up the data and validates to the same format. For this thepydanticlibrary is very helpful to constrain the Producer and Consumer data to be passed. This is how thefastapilibrary ensures data being passed around that API is validated and structured correctly. I wanted to use this pattern.ContributingIf you find this useful, consider adding a star and contributing.Currently this only uses thepika.BlockingChannelimplementation.TestsWhen running tests, a RabbitMQ instance needs to be up and running on your machine as the tests do live tests using that RabbitMQ.If using docker, you can spin up a RabbitMQ instance for testing usingdocker-compose -f docker-compose-rabbit.yml upThe environment variablePIKA_URLcan be overwritten to point to your test RabbitMQ instance.Then run tests usepytestpytest