Package: bufferq
bufferqBetter Queue Interface for PythonPython'squeueinterface is quite clunky and really not that good.bufferqis a separate queue implementation with a more intuitive interface.Sample UsageQueues frombufferqare simple to use with a simple interface:importthreadingimportbufferqq=bufferq.Queue()defconsumer(q):foriteminq.consume_one_generator():print(f"Working on item:{item}")thd=threading.Thread(target=consumer,args=(q,))thd.daemon=Truethd.start()q.put('a')q.put('b')time.sleep(1)q.put_multi(list(range(5)))q.stop()thd.join()What's Wrong withqueue?Here are a few issues:Design IssuesPython's queue does not provide much help for common tasks that queues are
used for, such as a list of work forProducer/Consumerdesign patterns.
Python's ownqueuedocumentation shows the followingexample:importthreading,queueq=queue.Queue()defworker():whileTrue:item=q.get()print(f'Working on{item}')print(f'Finished{item}')q.task_done()# turn-on the worker threadthreading.Thread(target=worker,daemon=True).start()# send thirty task requests to the workerforiteminrange(30):q.put(item)print('All task requests sent\n',end='')# block until all tasks are doneq.join()print('All work completed')Even here in the official docs, there are problems. They've omitted the
necessary exception handling; if the worker were to raise an exception betweenq.get()andq.task_done(), the call toq.join()might block indefinitely.
(Yes,print()is not likely to raise an exception, but real work done by such
a queueis...)
This can be fixed by adding try/finally, but the semantics are subtle and as
this example shows, error-prone.The example also does not actually terminate the consumer thread correctly and
instead just lets it die as a daemon thread. This might be okay for an example,
but this is not good for realistic uses of the queue where resources need
stricter management. This is doubly ironic, because the point ofq.join()is
to (presumably) supportdrainingthe queue and block until the queue is
empty. However, any logic to handle basic draining requires more tooling that
is outside of the queue (i.e. checking somethreading.Eventinstead ofwhile True), thus (in my opinion) defeating the point. This situation is
further complicated by the situation below:importqueueq=queue.Queue()q.put('a')defconsumer():q.get()# Uh-oh. No timeout argument passed, so this blocks indefinitely.# Ctrl+C to get out of this, or worse (!) since some versions of python# did not even support Ctrl+C in this setting...q.get()consumer()The consumer might be blocked waiting for an element before it has a chance to
check whether the stop event was set.Basic OperationsPython'squeueinterface is also a little sloppy for common operations,
like adding/pushing items to the queue.queue.Queue.put()has three
different arguments:The itemblocking (why?)timeout
(Options 2 and 3 are set so the operation blocks indefinitely until the
item can be added.)This is annoying; why haveblockingandtimeoutas separate arguments,
instead of simply lettingtimeout=0(or maybe even some placeholder-style
object if you are really,REALLYconcerned about blocking)? Atimeout=0should imply a single lookup that fails withqueue.Emptyif nothing is in
the queue without any additional arguments.
Yes, there is an added "convenience" call ofqueue.Queue.put_nowait(),
but this can just as easily be a proxy call to:put(item, timeout=0)which
can be added directly for clarity, but without muddying the rest of the
interface.This same problem exists (and is more relevant) for theget()calls for
the queue.Better DesignThe necessary variables to handle the draining should already implicitly be
available in the queue object, with improved calls. The queue should have somestop()call that stops the queue and wakes up anyone waiting indefinitely to
insert/remove an item with aQueueStopped()exception or similar to avoid
deadlock.Adding pythonic generators to remove items from the queue can also help with
these common cases; the consumer can simply iterate to obtain the next item
instead of handling the complicatedpop/getlogic that might otherwise be
required.This is all provided bybufferq.Queuelike below:importthreadingimportbufferqq=bufferq.Queue()defworker():foriteminq.consume_one_generator():print(f'Working on{item}')print(f'Finished{item}')# turn-on the worker threadthd=threading.Thread(target=worker,daemon=True).start()# send thirty task requests to the workerforiteminrange(30):q.put(item)print('All task requests sent, signal to stop and drain.')# Request that the queue stop, since everything has been added.q.stop()thd.join()print('All work completed and workers joined!')