Package: postgres-tq
Postgres Task QueueThis repo will contain thePostgresbased task queue logic, similar toour redis-tqbut based on postgres instead.Similar to redis-tq, this package allows for sharing data between multiple processes or hosts.Tasks support a "lease time". After that time other workers may consider this client to have crashed or stalled and pick up the item instead. The number of retries can also be configured.By default it keeps all the queues and tasks in a single tabletask_queue. If you want to use a different table for different queues for example it could also be configured when instantiating the queue.You can either set thecreate_table=Truewhen instantiating the queue or create the table yourself with the following query:CREATETABLEtask_queue(idUUIDPRIMARYKEY,queue_nameTEXTNOTNULL,taskJSONBNOTNULL,ttlINTNOTNULL,created_atTIMESTAMPNOTNULLDEFAULTCURRENT_TIMESTAMP,processingBOOLEANNOTNULLDEFAULTfalse,lease_timeoutFLOAT,deadlineTIMESTAMP,completed_atTIMESTAMP)Installationpostgres-tq is available onPyPIso you can simply install via:$pipinstallpostgres-tqHow it worksIt uses row level locks of postgres to mimic the atomic pop and atomic push of redis-tq when getting a new task from the queue:UPDATEtask_queueSETprocessing=true,deadline=current_timestamp+CAST(lease_timeout||' seconds'ASINTERVAL)WHEREid=(SELECTidFROMtask_queueWHEREcompleted_atISNULLANDprocessing=falseANDqueue_name=<your_queue_name>ANDttl>0ORDERBYcreated_atFORUPDATESKIPLOCKEDLIMIT1)RETURNINGid,task;Let's say two workers try to get a new task at the same time, assuming that they will probably see the same task to be picked up using the subquery:SELECTidFROMtask_queueWHEREcompleted_atISNULLANDprocessing=falseANDqueue_name=<your_queue_name>ANDttl>0ORDERBYcreated_atThe first worker locks the row with theFOR UPDATEclause until the update is completed and committed. If we hadn't used theSKIP LOCKEDclause, the second worker would have seen the same row and waited for the first worker to finish the update. However, since the first worker already updated it, the subquery would no longer be valid, and the second worker would return zero rows becauseWHERE id = NULL.However, since we are usingFOR UPDATE SKIP LOCKEDthe first worker locks the row for update and the second worker, skips that locked row and chooses another row for itself to update. This way we can avoid the race condition.The other methodscomplete()andreschedule()work similarly under the hood.How to useOn the producing side, populate the queue with tasks and a respective lease timeout:frompostgrestqimportTaskQueuetask_queue=TaskQueue(POSTGRES_CONN_STR,queue_name,reset=True)foriinrange(10):task_queue.add(some_task,lease_timeout,ttl=3)On the consuming side:frompostgrestqimportTaskQueuetask_queue=TaskQueue(POSTGRES_CONN_STR,queue_name,reset=True)whileTrue:task,task_id=task_queue.get()iftaskisnotNone:# do something with task and mark it as complete afterwardstask_queue.complete(task_id)iftask_queue.is_empty():break# task_queue.get is non-blocking, so you may want to sleep a# bit before the next iterationtime.sleep(1)Or you can even use the __iter__() method of the class TaskQueue and loop over the queue:frompostgrestqimportTaskQueuetask_queue=TaskQueue(POSTGRES_CONN_STR,queue_name,reset=True)fortask,id_intaskqueue:# do something with task and it's automatically# marked as completed by the iterator at the end# of the iterationIf the consumer crashes (i.e. the task is not marked as completed after lease_timeout seconds), the task will be put back into the task queue. This rescheduling will happen at most ttl times and then the task will be dropped. A callback can be provided if you want to monitor such cases.As the tasks are completed, they will remain in thetask_queuepostgres table. The table will be deleted of its content if
initializing aTaskQueueinstance with theresetflag totrueor if using theprune_completed_tasksmethod:frompostgrestqimportTaskQueue# If reset=True, the full queue content will be deletedtask_queue=TaskQueue(POSTGRES_CONN_STR,queue_name,reset=False)# Prune all tasks from queue completed more than 1 hour (in seconds)# ago. Tasks in progress, not started and completed recently will# stay in the postgres task_queue tabletask_queue.prune_completed_tasks(3600)Running the testsThe tests will check a presence of an Postgres DB in the port 15432. To initiate one using docker you can run:$makerun-postgresThen run$pdmrunmaketestpdm runwill ensure that themake testcommand is executed within the context of the virtual environment managed bypdm. Make sure you havepdmandDockerinstalled for this to work.LicenseThis project is licensed under theMIT License.