Package: persist-queue
persist-queueimplements a file-based queue and a serial of sqlite3-based queues. The goals is to achieve following requirements:Disk-based: each queued item should be stored in disk in case of any crash.Thread-safe: can be used by multi-threaded producers and multi-threaded consumers.Recoverable: Items can be read after process restart.Green-compatible: can be used ingreenletoreventletenvironment.Whilequeuelibandpython-pqueuecannot fulfil all of above. After some try, I found it’s hard to achieve based on their current
implementation without huge code change. this is the motivation to start this project.By default,persist-queueusepickleobject serialization module to support object instances.
Most built-in type, likeint,dict,listare able to be persisted bypersist-queuedirectly, to support customized objects,
please refer toPickling and unpickling extension types(Python2)andPickling Class Instances(Python3)This project is based on the achievements ofpython-pqueueandqueuelibSlack channelsJoinpersist-queuechannelRequirementsPython 2.7 or Python 3.x (refer toDeprecationfor future plan)Full support for Linux.Windows support (withCautionifpersistqueue.Queueis used).FeaturesMultiple platforms support: Linux, macOS, WindowsPure pythonBoth filed based queues and sqlite3 based queues are supportedFiled based queue: multiple serialization protocol support: pickle(default), msgpack, cbor, jsonDeprecationPython 3.4 release has reached end of lifeandDBUtilsceased support forPython 3.4,persist queuedrops the support for python 3.4 since version 0.8.0.
other queue implementations such as file based queue and sqlite3 based queue are still workable.Python 2 was sunset on January 1, 2020,persist-queuewill drop any Python 2 support in future version1.0.0, no new feature will be developed under Python 2.Installationfrom pypipip install persist-queue#formsgpack,cborandmysqlsupport,usefollowingcommandpip install persist-queue[extra]from source codegit clone https://github.com/peter-wangxu/persist-queue
cd persist-queue#formsgpackandcborsupport,run'pip install -r extra-requirements.txt'firstpython setup.py installBenchmarkHere are the time spent(in seconds) for writing/reading1000items to the
disk comparing the sqlite3 and file queue.WindowsOS: Windows 10Disk: SATA3 SSDRAM: 16 GiBWriteWrite/Read(1 task_done)Write/Read(many task_done)SQLite3 Queue1.88802.02903.5940File Queue4.95205.05608.4900windows notePerformance of Windows File Queue has dramatic improvement sincev0.4.1due to the
atomic renaming support(3-4X faster)LinuxOS: Ubuntu 16.04 (VM)Disk: SATA3 SSDRAM:  4 GiBWriteWrite/Read(1 task_done)Write/Read(many task_done)SQLite3 Queue1.82821.80752.8639File Queue0.91231.04112.5104Mac OSOS: 10.14 (macOS Mojave)Disk: PCIe SSDRAM:  16 GiBWriteWrite/Read(1 task_done)Write/Read(many task_done)SQLite3 Queue0.18790.21150.3147File Queue0.51580.53571.0446noteThe value above is in seconds for reading/writing1000items, the less
the betterAbove result was got from:python benchmark/run_benchmark.py 1000To see the real performance on your host, run the script underbenchmark/run_benchmark.py:python benchmark/run_benchmark.py <COUNT, default to 100>ExamplesExample usage with a SQLite3 based queue>>>importpersistqueue>>>q=persistqueue.SQLiteQueue('mypath',auto_commit=True)>>>q.put('str1')>>>q.put('str2')>>>q.put('str3')>>>q.get()'str1'>>>delqClose the console, and then recreate the queue:>>>importpersistqueue>>>q=persistqueue.SQLiteQueue('mypath',auto_commit=True)>>>q.get()'str2'>>>New functions:Available since v0.8.0shrink_disk_usageperform aVACUUMagainst the sqlite, and rebuild the database file, this usually takes long time and frees a lot of disk space afterget()Example usage of SQLite3 basedUniqueQThis queue does not allow duplicate items.>>>importpersistqueue>>>q=persistqueue.UniqueQ('mypath')>>>q.put('str1')>>>q.put('str1')>>>q.size1>>>q.put('str2')>>>q.size2>>>Example usage of SQLite3 basedSQLiteAckQueue/UniqueAckQThe core functions:put: add item to the queue. Returnsidget: get item from queue and mark as unack.  Returnsitem, Optional paramaters (block,timeout,id,next_in_order,raw)update: update an item. Returnsid, Paramaters (item), Optional parameter if item not in raw format (id)ack: mark item as acked. Returnsid, Parameters (itemorid)nack: there might be something wrong with current consumer, so mark item as ready and new consumer will get it.  Returnsid, Parameters (itemorid)ack_failed: there might be something wrong during process, so just mark item as failed. Returnsid, Parameters (itemorid)clear_acked_data: perform a sql delete agaist sqlite. It removes 1000 items, while keeping 1000 of the most recent, whose status isAckStatus.acked(note: this does not shrink the file size on disk) Optional paramters (max_delete,keep_latest,clear_ack_failed)shrink_disk_usageperform aVACUUMagainst the sqlite, and rebuild the database file, this usually takes long time and frees a lot of disk space afterclear_acked_dataqueue: returns the database contents as a Python List[Dict]active_size: The active size changes when an item is added (put) and completed (ack/ack_failed) unlikeqsizewhich changes when an item is pulled (get) or returned (nack).>>>importpersistqueue>>>ackq=persistqueue.SQLiteAckQueue('path')>>>ackq.put('str1')>>>item=ackq.get()>>># Do something with the item>>>ackq.ack(item)# If done with the item>>>ackq.nack(item)# Else mark item as `nack` so that it can be proceeded again by any worker>>>ackq.ack_failed(item)# Or else mark item as `ack_failed` to discard this itemParameters:clear_acked_datamax_delete(defaults to 1000): This is the LIMIT.  How many items to delete.keep_latest(defaults to 1000): This is the OFFSET.  How many recent items to keep.clear_ack_failed(defaults to False): Clears theAckStatus.ack_failedin addition to theAckStatus.ack.getraw(defaults to False): Returns the metadata along with the record, which includes the id (pqid) and timestamp.  On the SQLiteAckQueue, the raw results can be ack, nack, ack_failed similar to the normal return.id(defaults to None): Accepts anidor a raw item containingpqid.  Will select the item based on the row id.next_in_order(defaults to False): Requires theidattribute.  This option tells the SQLiteAckQueue/UniqueAckQ to get the next item based onid, not the first available.  This allows the user to get, nack, get, nack and progress down the queue, instead of continuing to get the same nack’d item over again.rawexample:>>>q.put('val1')>>>d=q.get(raw=True)>>>print(d)>>>{'pqid':1,'data':'val1','timestamp':1616719225.012912}>>>q.ack(d)next_in_orderexample:>>>q.put("val1")>>>q.put("val2")>>>q.put("val3")>>>item=q.get()>>>id=q.nack(item)>>>item=q.get(id=id,next_in_order=True)>>>print(item)>>>val2Note:The SQLiteAckQueue always uses “auto_commit=True”.The Queue could be set in non-block style, e.g. “SQLiteAckQueue.get(block=False, timeout=5)”.UniqueAckQonly allows for unique itemsExample usage with a file based queueParameters:path: specifies the directory wher enqueued data persisted.maxsize: indicates the maximum size stored in the queue, if maxsize<=0 the queue is unlimited.chunksize: indicates how many entries should exist in each chunk file on disk. When a all entries in a chunk file was dequeued by get(), the file would be removed from filesystem.tempdir: indicates where temporary files should be stored. The tempdir has to be located on the same disk as the enqueued data in order to obtain atomic operations.serializer: controls how enqueued data is serialized.auto_save:TrueorFalse. By default, the change is only persisted when task_done() is called. If autosave is enabled, info data is persisted immediately when get() is called. Adding data to the queue with put() will always persist immediately regardless of this setting.>>>frompersistqueueimportQueue>>>q=Queue("mypath")>>>q.put('a')>>>q.put('b')>>>q.put('c')>>>q.get()'a'>>>q.task_done()Close the python console, and then we restart the queue from the same path,>>>frompersistqueueimportQueue>>>q=Queue('mypath')>>>q.get()'b'>>>q.task_done()Example usage with an auto-saving file based queueAvailable since: v0.5.0By default, items added to the queue are persisted during theput()call,
and items removed from a queue are only persisted whentask_done()is
called.>>>frompersistqueueimportQueue>>>q=Queue("mypath")>>>q.put('a')>>>q.put('b')>>>q.get()'a'>>>q.get()'b'After exiting and restarting the queue from the same path, we see the items
remain in the queue, becausetask_done()wasn’t called before.>>>frompersistqueueimportQueue>>>q=Queue('mypath')>>>q.get()'a'>>>q.get()'b'This can be advantageous. For example, if your program crashes before finishing
processing an item, it will remain in the queue after restarting. You can also
spread out thetask_done()calls for performance reasons to avoid lots of
individual writes.Usingautosave=Trueon a file based queue will automatically save on every
call toget(). Callingtask_done()is not necessary, but may still be
used tojoin()against the queue.>>>frompersistqueueimportQueue>>>q=Queue("mypath",autosave=True)>>>q.put('a')>>>q.put('b')>>>q.get()'a'After exiting and restarting the queue from the same path, only the second item
remains:>>>frompersistqueueimportQueue>>>q=Queue('mypath',autosave=True)>>>q.get()'b'Example usage with a SQLite3 based dict>>>frompersisitqueueimportPDict>>>q=PDict("testpath","testname")>>>q['key1']=123>>>q['key2']=321>>>q['key1']123>>>len(q)2>>>delq['key1']>>>q['key1']Traceback(mostrecentcalllast):File"<stdin>",line1,in<module>File"persistqueue\pdict.py",line58,in__getitem__raiseKeyError('Key:{}not exists.'.format(item))KeyError:'Key: key1 not exists.'Close the console and restart the PDict>>>frompersisitqueueimportPDict>>>q=PDict("testpath","testname")>>>q['key2']321Multi-thread usage forSQLite3based queuefrompersistqueueimportFIFOSQLiteQueueq=FIFOSQLiteQueue(path="./test",multithreading=True)defworker():whileTrue:item=q.get()do_work(item)foriinrange(num_worker_threads):t=Thread(target=worker)t.daemon=Truet.start()foriteminsource():q.put(item)multi-thread usage forQueuefrompersistqueueimportQueueq=Queue()defworker():whileTrue:item=q.get()do_work(item)q.task_done()foriinrange(num_worker_threads):t=Thread(target=worker)t.daemon=Truet.start()foriteminsource():q.put(item)q.join()# block until all tasks are doneExample usage with a MySQL based queueAvailable since: v0.8.0>>>importpersistqueue>>>db_conf={>>>"host":"127.0.0.1",>>>"user":"user",>>>"passwd":"passw0rd",>>>"db_name":"testqueue",>>># "name": "",>>>"port":3306>>>}>>>q=persistqueue.MySQLQueue(name="testtable",**db_conf)>>>q.put('str1')>>>q.put('str2')>>>q.put('str3')>>>q.get()'str1'>>>delqClose the console, and then recreate the queue:>>>importpersistqueue>>>q=persistqueue.MySQLQueue(name="testtable",**db_conf)>>>q.get()'str2'>>>noteDue to the limitation of file queue described in issue#89,task_donein one thread may acknowledge items in other threads which should not be. Considering theSQLiteAckQueueif you have such requirement.Serialization via msgpack/cbor/jsonv0.4.1: Currently only available for file based Queuev0.4.2: Also available for SQLite3 based Queues>>>frompersistqueue>>>q=persistqueue.Queue('mypath',serializer=persistqueue.serializers.msgpack)>>># via cbor2>>># q = persistqueue.Queue('mypath', serializer=persistqueue.serializers.cbor2)>>># via json>>># q = Queue('mypath', serializer=persistqueue.serializers.json)>>>q.get()'b'>>>q.task_done()Explicit resource reclaimFor some reasons, an application may require explicit reclamation for file
handles or sql connections before end of execution. In these cases, user can
simply call:
.. code-block:: pythonq = Queue() # or q = persistqueue.SQLiteQueue(‘mypath’, auto_commit=True)
del qto reclaim related file handles or sql connections.Tipstask_doneis required both for file based queue and SQLite3 based queue (whenauto_commit=False)
to persist the cursor of nextgetto the disk.Performance impactWALStarting on v0.3.2, thepersistqueueis leveraging the sqlite3 builtin featureWALwhich can improve the performance
significantly, a general testing indicates thatpersistqueueis 2-4 times
faster than previous version.auto_commit=FalseSince persistqueue v0.3.0, a new parameterauto_commitis introduced to tweak
the performance for sqlite3 based queues as needed. When specifyauto_commit=False, user
needs to performqueue.task_done()to persist the changes made to the disk since
lasttask_doneinvocation.pickle protocol selectionFrom v0.3.6, thepersistqueuewill selectProtocol version 2for python2 andProtocol version 4for python3
respectively. This selection only happens when the directory is not present when initializing the queue.Testspersist-queueusetoxto trigger tests.Unit testtox -e <PYTHON_VERSION>Available<PYTHON_VERSION>:py27,py34,py35,py36,py37PEP8 checktox -e pep8pyenvis usually a helpful tool to manage multiple versions of Python.CautionCurrently, the atomic operation is supported on Windows while still in experimental,
That’s saying, the data inpersistqueue.Queuecould be in unreadable state when an incidental failure occurs duringQueue.task_done.DO NOT put any critical data on persistqueue.queue on Windows.ContributionSimply fork this repo and send PR for your code change(also tests to cover your change), remember to give a title and description of your PR. I am willing to
enhance this project with you :).LicenseBSDContributorsContributorsFAQsqlite3.OperationalError: database is lockedis raised.persistqueue open 2 connections for the db ifmultithreading=True, the
SQLite database is locked until that transaction is committed. Thetimeoutparameter specifies how long the connection should wait for the lock to go away
until raising an exception. Default time is10, increasetimeoutwhen creating the queue if above error occurs.sqlite3 based queues are not thread-safe.The sqlite3 queues are heavily tested under multi-threading environment, if you find it’s not thread-safe, please
make sure you set themultithreading=Truewhen initializing the queue before submitting new issue:).