Package: queue-utilities
queue-utilitiesLet's make using Queues great again! Queue utilities and conveniences for those using sync libraries.Currently implements using threads and threading queues as standard, multiprocessing queues can be used by passing in relevant multiprocessing.Queue arguments. Worker threads use threading.Thread not multiprocessing.Process by design, if you require running eg Select in an external process as a message broker I recommend spawning a Process and then using as is documented.This utilities package contains the following classes:Pipe- Pipe messages from one queue to another.Timer- Threaded timer that emits time on internal or provided queue after given wait time period. Can be cancelled.Ticker- Same as timer but emits time at regular intervals until stopped.Multiplex- Many-to-One (fan-in) queue management helper.Multicast- One-to-Many (fan-out) queue management helper.Select- Like Multiplex but output payload contains message source queue to be used in dynamic message based switching. Inspired by Golangs select statements using channels.as_thread- Decorator to run function in thread.with_input_queue- Decorator to attach input and optional output queues to function which will be run in a thread.with_output_queue- Decorator that sends function results to output queue.Note that this package is early stages of development.Installationpython3-mpipinstallqueue-utilitiesUsagePipefromqueue_utilitiesimportPipeoriginal_q,output_q=_queue.Queue(),_queue.Queue()p=Pipe(original_q,output_q)# put an item into the original queueoriginal_q.put(1)# get the message off the output queuerecv=output_q.get()print(recv)# 1# don't forget to stop the pipe after you've finished.p.stop()Timerfromqueue_utilitiesimportTimer# emit time after 5 secondsfive_seconds=Timer(5)five_seconds.get()# cancel a timerto_cancel=Timer(60)print(to_cancel._is_finished)# Falseto_cancel.stop()print(to_cancel._is_finished)# TrueTickerfromqueue_utilitiesimportTicker# print the time every 5 seconds 4 timestick=Ticker(5)for_inrange(4):print(f"The time is:{tick.get()}")# cancel the ticker threadtick.stop()Multiplexfromqueue_utilitiesimportMultiplexfromqueueimportQueue# create two queues and pass them into the Multiplexqueue_a,queue_b=Queue(),Queue()multi_p=Multiplex(queue_a,queue_b)# send messages to any of the queuesqueue_a.put("a")queue_b.put("b")# read the messagesfor_inrange(2):message=multi_p()# or multi_p.get()print(f"I got a message!{message}")# cleanupmulti_p.stop()# if you try to read a message after stop# it raises a MultiplexClosed exceptionmulti_p.get()Multicastfromqueue_utilitiesimportMulticastfromqueueimportQueueout_a,out_b=Queue(),Queue()multicast=Multicast(out_a,out_b)multicast.send("A message!")forqin(out_a,out_b):print(q.get())# prints "A message!" twice!multicast.stop()SelectUse with context to build a cancellable threadfromqueue_utilitiesimportSelectfromqueueimportQueuefromthreadingimportThreadout_a,cancel_sig=Queue(),Queue()defselector(*queues):withSelect(*queues)asselect:forwhich,messageinselect:ifwhichiscancel_sig:# stop select on any message on queue bselect.stop()else:print(f'Got a message{message}')Thread(target=selector,args=(out_a,cancel_sig)).start()out_a.put(1)out_a.put(2)out_a.put(3)cancel_sig.put('Bye')Timeout a function with TimerfromthreadingimportThreadimporttimefromqueueimportQueuefromqueue_utilitiesimportSelect,Timerdefdo_something_slow(q:Queue)->None:# do something slowtime.sleep(3)q.put("Done")output_q=Queue()Thread(target=do_something_slow,args=(output_q,)).start()timeout=Timer(2)select=Select(output_q,timeout._output_q)for(which_q,result)inselect:ifwhich_qisoutput_q:print("Received result",result)else:print("Function timed out!")breakselect.stop()as_threadfromqueue_utilitiesimportas_threadimporttime@as_threaddefsleeper():time.sleep(5)print('Goodbye!')sleeper()print('Waiting...')time.sleep(6)print('Done!')with_input_queuefromqueue_utilitiesimportwith_input_queuefromqueueimportQueuework_queue=Queue()result_queue=Queue()@with_input_queue(work_queue,result_queue)defsquarer(input:int):returninput**2foriinrange(10):work_queue.put(i)print(f'{i}squared is{result_queue.get()}')with_output_queuefromqueue_utilitiesimportwith_input_queuefromqueueimportQueueresult_queue=Queue()@with_output_queue(result_queue)defsquarer(input:int):returninput**2foriinrange(10):squarer(i)print(f'{i}squared is{result_queue.get()}')ContributingPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.LicenseMIT