Package: django-pgpubsub
django-pgpubsubdjango-pgpubsubprovides a framework for building an asynchronous
and distributed message processing network on top of a Django application
using a PostgreSQL database. This is achieved by leveraging Postgres'LISTEN/NOTIFYprotocol to build a message queue at the database layer.
The simple user-friendly interface,
minimal infrastructural requirements and the ability to leverage Postgres'
transactional behaviour to achieve exactly-once messaging, makesdjango-pgpubsuba solid choice as a lightweight alternative to AMPQ
messaging services, such asCeleryDocumentationhttps://django-pgpubsub.readthedocs.io/en/latest/Primary AuthorsPaul GilmartinWesley KendallHighlightsMinimal Operational Infrastructure: If you're already running a Django application
on top of a Postgres database, the installation of this library is the sum total
of the operational work required to implement a framework for a distributed
message processing framework. No additional frameworks or technologies
are required.Integration with Postgres Triggers (via django-pgtrigger):
To quote theofficialPostgres docs:"When NOTIFY is used to signal the occurrence of changes to a particular table,
a useful programming technique is to put the NOTIFY in a statement trigger that is triggered
by table updates.
In this way, notification happens automatically when the table is changed,
and the application programmer cannot accidentally forget to do it."By making use of thedjango-pgtriggerlibrary,django-pgpubsuboffers a Django application layer abstraction of the trigger-notify Postgres
pattern. This allows developers to easily write python-callbacks which will
be invoked (asynchronously) whenever a customdjango-pgtriggeris invoked.
Utilising a Postgres-trigger as the ground-zero for emitting a
message based on a database table event is far more robust than relying
on something at the application layer (for example, apost_savesignal,
which could easily be missed if thebulk_createmethod was used).Lightweight Polling: we make use of the PostgresLISTEN/NOTIFYprotocol to have achieve notification polling which usesno CPU and no database transactions unless there is a message to read.Exactly-once notification processing:django-pgpubsubcan be configured so
that notifications are processed exactly once. This is achieved by storing
a copy of each new notification in the database and mandating that a notification
processor must obtain a postgres lock on that message before processing it.
This allows us to have concurrent processes listening to the same message channel
with the guarantee that no two channels will act on the same notification. Moreover,
the use of Django's.select_for_update(skip_locked=True)method allows
concurrent listeners to continue processing incoming messages without waiting
for lock-release events from other listening processes.Durability and Recovery:django-pgpubsubcan be configured so that
notifications are stored in the database before they're sent to be processed.
This allows us to replay any notification which may have been missed by listening
processes, for example in the event a notification was sent whilst the listening
processes were down.Atomicity: The PostgresNOTIFYprotocol respects the atomicity
of the transaction in which it is invoked. The result of this is that
any notifications sent usingdjango-pgpubsubwill be sent if and only if
the transaction in which it sent is successfully committed to the database.Quick startPrerequisitesBefore using this library, you must be running Django 2.2 (or later) on top
of a (single) PostgreSQL 11 (or later) database.Installingpip install django-pgpubsubdjango-pgpubsubships with aNotificationmodel. This table must
be added to the app's database via the usual djangomigratecommand.
We should also addpgpubsubandpgtriggerintoINSTALLED_APPS.
Additionally, if we wish to run thepgpubsubtests, we need to addpgpubsub.testsintoINSTALLED_APPStoo.Minimal ExampleLet's get a brief overview of how to usepgpubsubto asynchronously
create aPostrow whenever anAuthorrow is inserted into the
database. For this example, our notifying event will come from a
postgres trigger, but this is not a requirement for all notifying events.
A more detailed version of this example, and an example which
does not use a postgres trigger, can be found in theDocumentation (by Example)section below.Define a ChannelChannels are the medium through which we send notifications.
We define our channel in our app'schannels.pyfile as a dataclass
as follows:fromdataclassesimportdataclassfrompgpubsub.channelimportTriggerChannelfrompgpubsub.tests.modelsimportAuthor@dataclassclassAuthorTriggerChannel(TriggerChannel):model=AuthorDefine a ListenerAlisteneris the function which processes notifications sent through a channel.
We define our listener in our app'slisteners.pyfile as follows:importdatetimeimportpgpubsubfrompgpubsub.tests.channelsimportAuthorTriggerChannelfrompgpubsub.tests.modelsimportAuthor,Post@pgpubsub.post_insert_listener(AuthorTriggerChannel)defcreate_first_post_for_author(old:Author,new:Author):print(f'Creating first post for{new.name}')Post.objects.create(author_id=new.pk,content='Welcome! This is your first post',date=datetime.date.today(),)Note that sinceAuthorTriggerChannelis a trigger-based channel, we need
to perform amigratecommand after first defining the above listener
so as to install the underlying trigger in the database.Finally, we must also ensure  that this listeners.py module is imported into the app's config
class. In this example, our app is calls "tests":# tests/apps.pyfromdjango.appsimportAppConfigclassTestsConfig(AppConfig):name='tests'defready(self):importpgpubsub.tests.listenersStart ListeningTo have our listener function listen for notifications on theAuthorTriggerChannel,
we use thelistenmanagement command:./manage.py listenNow whenever anAuthoris inserted into our database, our listener process creates
aPostobject referencing thatAuthor:https://user-images.githubusercontent.com/18212082/165683416-b5cbeca1-ea94-4cd4-a5a1-81751e1b0feb.movLive Demosbulk_createover several processesIn the below example we show howpgpubsubhandles a bulk creation
ofAuthorobjects when several processes are listening to theAuthorTriggerChannelchannel. For the sake of the below demonstration,
we added atime.sleep(3)statement into thecreate_first_post_for_authorlistener function. Note how only one processes is able to process any given
notification:https://user-images.githubusercontent.com/18212082/165823588-df91e84a-47f2-4220-8999-8556665e3de3.mov