Package: pgjobs
JobsA PL/PGSQL based work queue (Publisher/Consumer),
with a python asyncio/asyncpg apialpha softwareFeaturesImplements a two layer API:A postgresql layer: tasks can be published from PL/PGSQL functions,
or procedures. Also can be extended using triggers.A python layer (or any client with a postgresql driver). The default
implementations is based on asyncio python, using the awesome
asyncpg driver.It's compatible with postgrest. All procedures, and tables, are scoped
on an owned postgresql schema, and can be exposed throught it, with postgrestRetry logic, schedule_at or timeout, are implemented on the
publish method. A task, can be published, with a max_retries, param,
or an especific timeout.Internally uses two tablesjobs.job_queuethe table where pending and
running tasks are scheduled, andjobs.jobthe table where ended tasks,
are moved (success or failures).By default, tasks are retyried three times, with backoff.Timeout jobs, are expired, tasks by default had a 60s tiemout.Tasks can be scheduled on the future, just provide ascheduled_atparam.There are views to monitor queue stats:jobs.all(all tasks),jobs.expiredandjobs.runningTasks could also be priorized, provide a priority number, greater priority,
precedence over other tasksconsumer_topic, allows to consume tasks with a * (topic.element.%)rudimentary benchs on my laptop showed that it can handle 1000 tasks/second,
but anyway it depends on your postgres instance.instead of a worker daemon, tasks could also be consumed from a cronjob, or
a regular python or a kubernetes job. (It could be used to parallelize k8 jobs)tradeofsAll jobs had to be aknowledged positive or negative (ack/nack)Use from postgresqlSELECTjob_idFROMjobs.publish(i_task-- method or function to be executed,i_body::jsonb=null-- arguments passed to it (on python {args:[], kwargs:{}}),i_scheduled_at:timestamp=null,-- when the task should runi_timeout:numeric(7,2)-- timeout in seconds for the jobi_priority:integer=null-- gretare number more priority)On the worker side:SELECT*fromjobs.consume(num:integer-- number of desired jobs);returns a list of jobs to be processed,Or selective consume a topic:SELECT*fromjobs.consume_topic('topic.xxx.%',10)jobs are marked as processing,
and should be acnlowledged with:SELECTFROMjobs.ack(job_id);or to return a failed job.SELECTFROMjobs.nack(job_id,traceback,i_schedule_at)Also you can batch enqueue multiple jobs in a single request, usingSELECT*FROMjobs.publish_bulk(jobs.bulk_job[]);where jobs.bulk_job iscreatetypejobs.bulk_jobas(taskvarchar,bodyjsonb,scheduled_attimestamp,timeoutinteger,priorityinteger,max_retriesinteger);Use from pythonOn this side, implementing a worker, should be something likedb=awaitasyncpg.connect(dsn)whileTrue:jobs=awaitjobs.consume(db,1)forjobinjobs:try:awaitjobs.run(db,job["job_id"])awaitjobs.ack(job["job_id"])exceptExceptionase:awaitjobs.nack(job["job_id"],str(e))awaitasyncio.sleep(1)On the publisher side, jobs could be enqueued from between a
postgresql transaction:db=awaitasyncpg.connect(dsn)asyncwithdb.transaction():# do whatever is needed,# queue a taskawaitjobs.publish("package.file.sum",args=[1,2])Installing the packagepipinstallpgjobs
jobs-migratepostgresql://user:password@localhost:5432/db

Thiswillcreatetheschemaonthe`jobs`postgresqlschemaTo run the worker,jobs-worker postgresql://dsnAt the moment there are no too much things implemented there,
just a single threaded worker, that needs a bit more of love :)
If your application resides on a python package,
tasks likeyourpackage.file.methodwill be runnable as is.Observavility and monitorWith psql, or exposing them throught postgresql_exporterTODOconnect notifications, using pg_notify, when tasks are queued,
are picked, are completed. With this in place, it's easy
enought to write o WS to send notifications to connected customers.improve the worker to run every job on an asyncio taskhandle better exceptions on the python sidefix requirements fileadd github actions to run CIwrite better docs and some examples