Package: pq
A transactional queue system for PostgreSQL written in Python.It allows you to push and pop items in and out of a queue in various
ways and also provides two scheduling options: delayed processing and
prioritization.The system uses a single table that holds all jobs across queues; the
specifics are easy to customize.The system currently supports only thepsycopg2database driver - orpsycopg2cffifor PyPy.The basic queue implementation is similar to Ryan Smith’squeue_classiclibrary written in Ruby, but usesSKIP LOCKEDfor concurrency control.In terms of performance, the implementation clock in at about 1,000
operations per second. Using thePyPyinterpreter, this scales linearly with the number of cores available.Getting startedAll functionality is encapsulated in a single classPQ.class PQ(conn=None, pool=None,table="queue",schema=None)The optionalschemaargument can be used to qualify the table with
a schema if necessary.Example usage:frompsycopg2importconnectfrompqimportPQconn=connect('dbname=example user=postgres')pq=PQ(conn)For multi-threaded operation, use a connection pool such aspsycopg2.pool.ThreadedConnectionPool.You probably want to make sure your database is created with theutf-8encoding.To create and configure the queue table, call thecreate()method.pq.create()QueuesThepqobject exposes queues through Python’s dictionary
interface:queue=pq['apples']Thequeueobject providesgetandputmethods as explained
below, and in addition, it also works as a context manager where it
manages a transaction:withqueueascursor:...The statements inside the context manager are either committed as a
transaction or rejected, atomically. This is useful when a queue is
used to manage jobs because it allows you to retrieve a job from the
queue, perform a job and write a result, with transactional
semantics.MethodsUse theput(data)method to insert an item into the queue. It
takes a JSON-compatible object such as a Python dictionary:queue.put({'kind':'Cox'})queue.put({'kind':'Arthur Turner'})queue.put({'kind':'Golden Delicious'})Items are pulled out of the queue usingget(block=True). The
default behavior is to block until an item is available with a default
timeout of one second after which a value ofNoneis returned.defeat(kind):print'umm,%sapples taste good.'%kindjob=queue.get()eat(**job.data)Thejobobject provides additional metadata in addition to thedataattribute as illustrated by the string representation:>>> job
<pq.Job id=77709 size=1 enqueued_at="2014-02-21T16:22:06Z" schedule_at=None>Thegetoperation is also available through iteration:forjobinqueue:ifjobisNone:breakeat(**job.data)The iterator blocks if no item is available. Again, there is a default
timeout of one second, after which the iterator yields a value ofNone.An application can then choose to break out of the loop, or wait again
for an item to be ready.forjobinqueue:ifjobisnotNone:eat(**job.data)# This is an infinite loop!SchedulingItems can be scheduled such that they’re not pulled until a later
time:queue.put({'kind':'Cox'},'5m')In this example, the item is ready for work five minutes later. The
method also acceptsdatetimeandtimedeltaobjects.PriorityIf some items are more important than others, a time expectation can
be expressed:queue.put({'kind':'Cox'},expected_at='5m')This tells the queue processor to give priority to this item over an
item expected at a later time, and conversely, to prefer an item with
an earlier expected time. Note that items without a set priority are
pulled last.The scheduling and priority options can be combined:queue.put({'kind':'Cox'},'1h','2h')This item won’t be pulled out until after one hour, and even then,
it’s only processed subject to it’s priority of two hours.Encoding and decodingThe task data is encoded and decoded into JSON using the built-injsonmodule. If you want to use a different implementation or need
to configure this, passencodeand/ordecodearguments to thePQconstructor.PicklesIf a queue name is provided as<name>/pickle(e.g.'jobs/pickle'), items are automatically pickled and
unpickled using Python’s built-incPicklemodule:queue=pq['apples/pickle']classApple(object):def__init__(self,kind):self.kind=kindqueue.put(Apple('Cox'))This allows you to store most objects without having to add any
further serialization code.The old pickle protocol0is used to ensure the pickled data is
encoded asasciiwhich should be compatible with any database
encoding. Note that the pickle data is still wrapped as a JSON string at the
database level.While using the pickle protocol is an easy way to serialize objects,
for advanced users t might be better to use JSON serialization
directly on the objects, using for example the object hook mechanism
in the built-injsonmodule or subclassingJSONEncoder <https://docs.python.org/2/library/json.html#json.JSONEncoder>.Taskspqcomes with a higher levelAPIthat helps to managetasks.frompq.tasksimportPQpq=PQ(...)queue=pq['default']@queue.task(schedule_at='1h')defeat(job_id,kind):print'umm,%sapples taste good.'%kindeat('Cox')queue.work()tasks’sjobscan optionally be re-scheduled on failure:@queue.task(schedule_at='1h',max_retries=2,retry_in='10s')defeat(job_id,kind):# ...Time expectations can be overriden attaskcall:eat('Cox',_expected_at='2m',_schedule_at='1m')** NOTE ** First positional argument is id of job. It’s PK of record in PostgreSQL.Thread-safetyAll objects are thread-safe as long as a connection pool is provided
where each thread receives its own database connection.Changes1.9.1 (2023-04-04)Add support for PostgreSQL 14 [kalekseev].Seehttps://www.postgresql.org/docs/14/release-14.html#id-1.11.6.11.4.Add support for using a custom schema (issue #35).1.9.0 (2020-09-29)The task executor now receivesjob_idas the first argument.1.8.2 (2020-08-14)Added support for queue names longer than 63 characters.A database migration (dropping and recreating thepq_notifytrigger) is required if using names longer than this limit. If not
using, then no migration is required.Return connections to the pool if an exception is raised while it is retrieved1.8.1 (2019-07-30)Added overridableencodeanddecodemethods which are
responsible for turning task data intoJSONand vice-versa.1.8.0 (2019-07-03)Change policy on task priority. Tasks with a null-value forexpected_atare now processed after items that have a value set.1.7.0 (2019-04-07)UseSKIP LOCKEDinstead of advisory lock mechanism (PostgreSQL 9.5+).1.6.1 (2018-11-14)Fix queue class factory pattern.1.6 (2018-11-12)Fix compatibility withNamedTupleCursor.Fix duplicate column name issue.Add option to specify own queue class.1.5 (2017-04-18)Fixed Python 2 compatibility.1.4 (2016-03-25)Added worker class and handler helper decorator.
[jeanphix]1.3 (2015-05-11)Python 3 compatibility.
[migurski]Fix time zone issue.1.2 (2014-10-21)Improvements:Fixed concurrency issue where a large number of locks would be held
as a queue grows in size.Fixed a database connection resource issue.1.1 (2014-02-27)Features:A queue is now also a context manager, providing transactional
semantics.A queues now returns task objects which provide metadata and allows
reading and writing task data.Improvements:The same connection pool can now be used with different queues.Bugs:TheLiteralstring wrapper did not work correctly withpsycopg2.The transaction manager now correctly returns connections to the
pool.1.0 (2013-11-20)Initial public release.