Package: quick-queue
Quick Multiprocessing QueueThis is an implementation of Quick Multiprocessing Queue for Python and work similar tomultiprocessing.queue(more
information aboutmultiprocessing.queueinhttps://docs.python.org/3/library/multiprocessing.html?highlight=process#pipes-and-queues).InstallLast release version of the project to install in:https://pypi.org/project/quick_queue_project/pip install quick-queueIntroductionThe motivation to create this class is due tomultiprocessing.queueis too slow putting and getting elements
to transfer data between python processes.But if you put or get one list with elements work similar as put or get one single element; this list is getting as
fast as usually but this has too many elements for process in the subprocess and this action is very quickly.In other words, Multiprocess queue is pretty slow putting and getting individual data, then QuickQueue wrap several
data in one list, this list is one single data that is enqueue in the queue than is more quickly than put one
individual data.While Producer produce and put lists of elements in queue, subprocesses consume those lists and iterate every element,
then subprocesses have elements very quickly.Quick useQuickQueueImport:fromquick_queueimportQQueuePseudocode without process:qq=QQueue()# << Add here `qq` to new process(es) and start process(es) >>qq.put("value")# Put all the values you needqq.end()# When end put values call to end() to mark you will not put more values and close QQueueComplete example (it needsimport multiprocessing):def_process(qq):print(qq.get())print(qq.get())print(qq.get())if__name__=="__main__":qq=QQueue()p=multiprocessing.Process(target=_process,args=(qq,))p.start()qq.put("A")qq.put("B")qq.put("C")qq.end()p.join()Note: you need to callendmethod to perform remain operation and close queue. If you only want put remain data in
queue, you can callput_remain, then you need to call manually toclose(orend, this performscloseoperation
too).You can put all values in one iterable or several iterables withput_iterablemethod (put_iterableperform remain
operation when iterable is consumed; but this not close queue, you need call toclose()or toend()in this case):def_process(qq):print(qq.get())print(qq.get())print(qq.get())if__name__=="__main__":qq=QQueue()p=multiprocessing.Process(target=_process,args=(qq,))p.start()qq.put_iterable(["A","B","C"])qq.put_iterable(["D","E","F"])qq.end()p.join()If you need to useputin other process, then you need to initialize values in QQueue withinit. Due to
Python message pass between process it is not possible share values in the same shared Queue object (at least I have
not found the way) and, by other side, maybe you want to define a different initial values per "put process" to
sensor work calculation.def_process(qq):# Define initial args to this process, if you do not call to init method, then it use default valuesqq.init("""<Defined args>""")qq.put("A")qq.put("B")qq.put("C")qq.end()if__name__=="__main__":qq=QQueue()p=multiprocessing.Process(target=_process,args=(qq,))p.start()print(qq.get())print(qq.get())print(qq.get())p.join()You can use defined args in the main constructor if you pass values. You can get initial args
withget_init_args(return a dict with your args) in process where you instanced QQueue,
then in second process you can expand those args ininitmethod with**.def_process(qq,init_args):qq.init(**init_args)qq.put("A")qq.put("B")qq.put("C")qq.end()if__name__=="__main__":qq=QQueue("""<Defined args>""")p=multiprocessing.Process(target=_process,args=(qq,qq.get_init_args()))p.start()print(qq.get())print(qq.get())print(qq.get())p.join()QuickJoinableQueueYou can use a Joinable Queue if you want usejoinandtask_donein queue.Import:fromquick_queueimportQJoinableQueuePseudocode without process:qjq=QJoinableQueue()# << Add here `qq` to new process(es) and start process(es) >>qjq.put("value")# Put all the values you needqjq.join()# When end put values call to put_remain() or join() to mark you will not put more values QJoinableQueue.qjq.close()# You can close QJoinableQueue with close()Complete example (it needsimport multiprocessing):def_process(qjq):print(qjq.get())qjq.task_done()print(qjq.get())qjq.task_done()print(qjq.get())qjq.task_done()if__name__=="__main__":qjq=QJoinableQueue()p=multiprocessing.Process(target=_process,args=(qjq,))p.start()qjq.put("A")qjq.put("B")qjq.put("C")qjq.join()qjq.close()p.join()Note: withjoinyou have not call toendbecause this close the queue.joinmethod perform remain operation
(put_remain) but not close the queue or callput_remaindirectly if you need it. You need to call manually toclosebecause afterjoinyou can to do other operations with this queue.You can put all values in one iterable or several iterables withput_iterablemethod (put_iterableperform remain
operation when iterable is consumed; but this not close queue, you need call toclose()in this case):def_process(qjq):print(qjq.get())qjq.task_done()print(qjq.get())qjq.task_done()print(qjq.get())qjq.task_done()if__name__=="__main__":qjq=QJoinableQueue()p=multiprocessing.Process(target=_process,args=(qjq,))p.start()qjq.put_iterable(["A","B","C"])qjq.put_iterable(["D","E","F"])qjq.join()qjq.close()p.join()If you need to useputin other process, then you need to initialize values in QJoinableQueue withinit.def_process(qjq):# Define initial args to this process, if you do not call to init method, then it use default valuesqjq.init("""<Defined args>""")qjq.put("A")qjq.put("B")qjq.put("C")qjq.join()if__name__=="__main__":qjq=QJoinableQueue()p=multiprocessing.Process(target=_process,args=(qjq,))p.start()print(qjq.get())qjq.task_done()print(qjq.get())qjq.task_done()print(qjq.get())qjq.task_done()p.join()You can use defined args in the main constructor if you pass values.def_process(qjq,init_args):qjq.init(**init_args)qjq.put("A")qjq.put("B")qjq.put("C")qjq.join()if__name__=="__main__":qjq=QJoinableQueue("""<Defined args>""")p=multiprocessing.Process(target=_process,args=(qjq,qjq.get_init_args()))p.start()print(qjq.get())qjq.task_done()print(qjq.get())qjq.task_done()print(qjq.get())qjq.task_done()p.join()About performanceAn important fact is the size of list (named here "bucket list") in relation producer and consumers process to have
the best performance:If queue is full, mean consumers are slower than producer.If queue is empty, mean producer is slower than consumers.Then, best size of bucket list (size_bucket_list) is where queue is not full and not empty; for this, I implemented
one sensor to determinate in realtime thesize_bucket_list, you can enable this sensor ifsize_bucket_listisNone(if you define a number insize_bucket_list, then you want a constant value tosize_bucket_listand sensor
disable). by default sensor is enabled (size_bucket_list=None), because depend on Hardware in your computer thissize_bucket_listvalue should change, I recommend you test the best performance for your computer modifyingsize_bucket_list(withNoneand with number value).You can delimit sensor scope withmin_size_bucket_listandmax_size_bucket_list(ifmax_size_bucket_listis None then is infinite):qq=QQueue(min_size_bucket_list=10,max_size_bucket_list=1000)To disable the sensor define a size insize_bucket_list:qq=QQueue(size_bucket_list=120)Performance testHardware where the tests have been done:Processor: Intel i5 3.2GHzOperating System: Windows 10 x64Queue vs QuickQueueUsepython3 tests\performance_qqueue_vs_queue.pyPut in a producer process and get in a consumer process N elements withQuickQueueandmultiprocessing.queue:10,000,000 elements (time: Queue = QuickQueue x 13.28 faster):QuickQueue: 0:00:24.436001 | Queue: 0:05:24.4881491,000,000 elements (time: Queue = QuickQueue x 17.55 faster):QuickQueue: 0:00:01.877998 | Queue: 0:00:32.951001100,000 elements (time: Queue = QuickQueue x 6.32 faster):QuickQueue: 0:00:00.591002 | Queue: 0:00:03.736011JoinableQueue vs JoinableQueueUsepython3 tests\performance_qjoinablequeue_vs_joinablequeue.pyPut in a producer process and get in a consumer process N elements withQuickJoinableQueueandmultiprocessing.JoinableQueue:10,000,000 elements (time: JoinableQueue = QuickJoinableQueue x 6.12 faster):QuickJoinableQueue: 0:01:10.113079 | JoinableQueue: 0:08:09.9745701,000,000 elements (time: JoinableQueue = QuickJoinableQueue x 7.05 faster):QuickJoinableQueue: 0:00:06.858233 | JoinableQueue: 0:00:48.363999100,000 elements (time: JoinableQueue = QuickJoinableQueue x 3.33 faster):QuickJoinableQueue: 0:00:01.192382 | JoinableQueue: 0:00:03.702002DocumentationFunctions:QQueue: Main method to create aQuickQueueobject configured. Args:maxsize: maxsize of bucket lists in queue. Ifmaxsize<=0then queue is infinite (and sensor is disabled, I
recommend always define one positive number to save RAM memory). By default:1000size_bucket_list:Noneto enable sensor size bucket list (requiremaxsize>0). If a number is defined
here then use this number to size_bucket_list and disable sensor. Ifmaxsize<=0andsize_bucket_list==Nonethen size_bucket_list is default to1000;other wise,
if maxsize<=0 and size_bucket_list is defined, then use this number. By default:Nonemin_size_bucket_list: (only if sensor is enabled) min size bucket list.Min == 1andmax == max_size_bucket_list - 1. By default:10max_size_bucket_list: (only if sensor is enabled) max size bucket list. IfNoneis infinite.
By default:NoneQJoinableQueue: Main method to create aQuickJoinableQueueobject configured. Args:maxsize: maxsize of bucket lists in queue. Ifmaxsize<=0then queue is infinite (and sensor is disabled, I
recommend always define one positive number to save RAM memory). By default:1000size_bucket_list:Noneto enable sensor size bucket list (requiremaxsize>0). If a number is defined
here then use this number to size_bucket_list and disable sensor. Ifmaxsize<=0andsize_bucket_list==Nonethen size_bucket_list is default to1000;other wise,
if maxsize<=0 and size_bucket_list is defined, then use this number. By default:Nonemin_size_bucket_list: (only if sensor is enabled) min size bucket list.Min == 1andmax == max_size_bucket_list - 1. By default:10max_size_bucket_list: (only if sensor is enabled) max size bucket list. IfNoneis infinite.
By default:NoneClass:QuickQueueThis is a class with heritagemultiprocessing.queues.Queue. Methods overwritten:put_bucket: This put in the queue a list of data.put: This put in the queue a data wrapped in a list. Accumulate data until size_bucket_list, then put in queue.put_remain: Call to enqueue rest values that remains.put_iterable: This put in this QQueue all data from an iterable.end: Helper to call to put_remain and close queue in one method.get_bucket: This get from queue a list of data.get: This get from queue a data unwrapped from the list.qsize: This return the number of bucket lists (not the number of elements)QuickJoinableQueueThis is a class with heritageQuickQueueandmultiprocessing.queues.JoinableQueue. Methods overwritten:put_bucket: This put in the queue a list of data.join: This call toput_remainand call tojoin(Wait until the thread terminates) frommultiprocessing.queues.JoinableQueue.end: Raise a warning for bad use andput_remainredefined.Not overwritten but it is important for this class:task_done: Indicate that a formerly enqueued task is complete.ImprovementsTo implementQuickJoinableQueueI need to call toreleaseSemaphore one time for each element of bulk, this is not
the best solution, but it is the easy way to implementJoinableQueuewith security.Is useful for you?Maybe you would be so kind to consider the amount of hours puts in, the great effort and the resources expended in
doing this project. Thank you.