Package: asyncpg-queue
asyncpg-queuePostgres (asynchronous) queuesUse Postgres to manage Python workloads asynchronously, powered byasyncpg. asyncpg-queue is a simple library whose features include:Ability to run both synchronous and asynchronous Python callablesAt-least-once execution of queued tasksScales with the number of database connections availableDependency free apart fromasyncpgUses Postgres notification channels to not thrash the database with unnecessary pollingUsageTo get started using asyncpg-queue, initialize the Postgres objects that it relies on:importasyncpgfromasyncpg_queueimportbootstrapdb=asyncpg.connect("postgresql://postgres@127.0.0.1:5432/postgres")awaitbootstrap(db)Now tasks can be enqueued for future processing. Thequeue.putmethod is naive and
should, in most cases be used within a transaction like in the following contrived
example:fromasyncpg_queueimportqueuedb=asyncpg.connect("postgresql://postgres@127.0.0.1:5432/postgres")asyncwithdb.transaction():awaitdb.execute("INSERT INTO users (name, email) VALUES ($1, $2)","Someone Like a User","someone@example.com",)awaitqueue.put(db,"send-welcome-email",data={"email":"someone@example.com","name":"Someone Like a User","stuff":"more of it"},)The utility of usingputwithin a transaction is that often tasks that are meant to
be processed asynchronously should only be enqueued if the generating process succeeds.
The above relies on the database transaction successfully being committed as a strong
indicator that the user was successfully created and therefore should receive a welcome
email. However, there is no requirement thatputmust be called within a transaction.Processing tasks entails creating and running a worker process.fromasyncpg_queueimportWorkerdefsend_welcome_email(email,name,**kwargs):print("sending a welcome email!")worker=Worker("postgresqlL//postgres@127.0.0.1:5432/postgres",tasks={"send-welcome-email":send_welcome_email,})awaitworker.run()Notice thetasksparameter passed as part ofWorker's initialization. This map
instructs the worker to process the "send-welcome-email" queue of tasks with the
specified function.Contributingasyncpg-queue usesPoetryto manage its
dependencies, development tooling, and buiild.poetryinstallTestingA Docker container running Postgres is used during testing. Assuming thatdockeris
available on your system path at the time of running tests, the appropriate image(s)
will be pulled.Tests are invoked by Pytest:poetryrunpytesttest/Alternatively, if you have a running Postgres instance and do not want to rely on
Docker, pass the DSN of a running Postgres instance that can be used during testing:poetryrunpytest--postgres-dsn=postgresql://postgres@localhost:5433/postgrestest/FormattingCode formatting is enforced byBlack:poetryrunblack.Static analysisLinting (and auto-fixing where possible) is done byRuff:poetryrunruffcheck--fix.Types are checked withMypy:poetryrunmypy--install-types./asyncpg_queue/./test/Unused code is checked byVulture:poetryrunvultureasyncpg_queue/test/MotivationKeep your project simple as long as possible! While simplicity is in the eye of the
beholder, the definition used here amounts to, refrain from adding additional tools
until necessary.Many projects begin simply with a server and a data-store. Eventually, as the project
gains users and gathers complexity there may be a need for doingsomethingin a
separate process so as to not impede the main line. This something could be sending
emails by poking some email SaaS provider's API or calculating the total number of new
users of a particular feature at the end of the day. asyncpg-queue is meant for this
moment in an application's history.asyncpg-queue and similar implementations have been successfully used to prolong or
forestall implementing queues and background workers with Redis, Celery or a variety
of other data stores. While many of these tools are not difficult to operate and PaaS
vendors often have a managed version, there is always an additional complexity cost
from introducing a new tool. asyncpg-queue should keep your toolset consistent since
it only relies on Python and Postgres.Whennotto useThe primary caveat of this library is that if the database is the bottleneck in an
application deployment then using this tool will only add to the pressure on Postgres.
There will be more connections opened, more queries, and some additional data stored.
If any of those areas are problems, they will almost undoubtedly get worse with the
introduction of asyncpg-queue.While fast enough, asyncpg-queue has little ability to ramp the performance of
producers (ie: adding to the queue) or consumers (ie: popping from the queue) because
of its reliance on Postgres. Only so much data can be written or read given a network
configuration and the server instance running Postgres. To play around with this idea
consultexample/benchmark/producer.pyandexample/benchmark/consumer.pywhich should provide
estimates of the maximum read and write throughput of your setup.asyncpg-queue is well suited for workloads that are mostly I/O. An example would be
calculating an end-of-day rollup table in Postgres that takes a long time to run.
However, asyncpg-queue is ill suited for running many CPU intensive tasks, like
training a neural network or performing the same end-of-day rollup in memory. In these
cases it's necessary to pay attention to theconcurrencyparameter ofWorker.