Package: ezq
ezqSimple wrappers for python multiprocessing and threading.Changelog-Issues-DocumentationWhy?ezqmakes it easy to connect subprocesses and threads (both considered "workers") using queues with a simpler API thanconcurrent.futures,multiprocessing, orthreading.Installpython-mpipinstallezqExample: Quick StartIf you just want to apply a function to some inputs, you can useezq.map()to run it on all available CPUs and get the results back.importezqprint(list(ezq.map(lambdax:x*2,range(6))))# => [0, 2, 4, 6, 8, 10]Example: Sum MessagesHere's a simple example of a worker that reads from an input queue, sums up the messages, and puts the result on an output queue.importezqdefworker(q,out):"""Add up all the messages."""total=0formsginq:# read a message from the queuetotal+=msg.data# after reading all the messages, write the totalout.put(total)defmain():"""Run several workers."""# Step 1: Creates the queues and start the workers.q,out=ezq.Q(),ezq.Q()# input & output queuesworkers=[ezq.run(worker,q,out)for_inrange(ezq.NUM_CPUS)]# workers are all running# Step 2: Send work to the workers.foriinrange(1000):q.put(i)# send work# Step 3: Tell the workers to finish.q.stop(workers)# workers are all stopped# Step 4: Process the results.want=sum(range(1000))have=sum(msg.dataformsginout.items())asserthave==wantprint(have)if__name__=="__main__":main()Typical worker lifecycleThe main processcreates queueswithezq.Q.The main processcreates workerswithezq.run(alias forWorker.process) orezq.run_thread(alias forWorker.thread).The main processsends datausingQ.put.The workeriterates over the queue.The main processends the queuewithQ.stop.The worker returns when it reaches the end of the queue.(Optional) The main processprocesses the results.ProcessvsThreadezqsupports two kinds of workers:ProcessandThread. There is a lot of existing discussion about when to use which approach, but a general rule of thumb is:Processis forparallelismso you can use multiple CPUs at once. Ideal forCPU-boundtasks like doing lots of mathematical calculations.Threadis forconcurrencyso you can use a single CPU to do multiple things. Ideal forI/O-boundtasks like waiting for a disk, database, or network.Some more differences:Shared memory: EachProcessworker hasdata sent to it viapickle(actuallydill, apicklereplacement) and it doesn't share data with other workers. By contrast, eachThreadworker shares its memory with all other workers on the same CPU, so it canaccidentally change global state.Queue overhead:ezq.Qhas more overheadforProcessworkers thanThreadworkers.Creating sub-workers:ProcessandThreadworkers can create additionalThreadworkers, butthey cannot create additionalProcessworkers.Create queuesIn the main process, create the queues you'll need. Here are my common situations:0 queues: I'm using a simple function and can askezq.mapto make the queues for me.1 queue: the worker reads from an input queue and persists the result somewhere else (e.g., writing to disk, making a network call, running some other program).2 queues(most common): the worker reads from an input queue and write the results to an output queue.3 queues: multiple stages of work are happening where workers are reading from one queue and writing to another queue for another worker to process.NOTE:If you're usingThreadworkers, you can save some overhead by passingQ("thread"). This lightweight queue also doesn't usepickle, so you can use it to pass hard-to-pickle things (e.g., database connection).q,out=ezq.Q(),ezq.Q()# most commonq2=ez.Q("thread")# only ok for Thread workersA worker task is just a functionIn general, there's nothing special about a worker function, but note:If you're usingProcessworkers, all arguments arepassed throughpicklefirst.We don't currently do anything with the return value of this function (unless you useezq.map()). You'll need an output queue to return data back to the main process/thread.Create workersIn the main process, create workers usingezq.runorezq.run_threadwhich take a function and any additional parameters. Typically, you'll pass the queues you created to the workers at this point.NOTE:ProcessandThreadworkers can create additionalThreadworkers, butthey cannot create additionalProcessworkers.Send dataOnce you've created the workers, you send them data withQ.putwhich createsezq.Msgobjects and puts them in the queue. Each message has three attributes (all optional):data: Any- This is the data you want the worker to work on.kind: str- You can use this to send multiple kinds of work to the same worker. Note that the specialENDkind is used to indicate the end of a queue.order: int- This is the message order which can help you reorder results or ensure that messages from a queue are read in a particular order (that's whatQ.sorted()uses).BewarepickleIf you are usingProcessworkers, everything passed to the worker (arguments, messages) is first passed topickle(actually,dill). Anything that cannot be pickled with dill (e.g., database connections), cannot be passed toProcessworkers. Note thatdillcanserialize many more types thanpickle(e.g.lambdafunctions).Beware shared stateIf you are usingThreadworkers, workers can share certain variables, so you need to be careful of how variables are access to avoid accidentally corrupting data.Iterate over messagesInside the worker, iterate over the queue to read each message until the queue ends (see below). If the messages need to be processed in order, useQ.sorted.formsginq:# read each message until the queue ends...formsginq.sorted():# read each message in order...End the queueAfter the main process has sent all the data to the workers, it needs to indicate
that there's no additional work to be done. This is done by callingQ.stop()using the input queue that the workers are reading from and passing the list of workers to wait for.In some rare situations, you can useQ.end()to explicitly end the queue.Process resultsIf you have an output queue, you may want to to process the results. You can useQ.items()to end the queue and read the current messages.importezqout=ezq.Q()...result=[msg.dataformsginout.items()]# ORresult=[msg.dataformsginout.items(sort=True)]# sorted by Msg.order# ORresult=[msg.dataformsginout.items(cache=True)]# cache the messagesExample: Read and Write QueuesIn this example, several workers read from a queue, process data, and then write to a different queue that a single worker uses to print to the screen sorting the results as it goes along.Note that we use a singlewriterto avoid clashes or overwriting.importezqdefprinter(out:ezq.Q)->None:"""Print results in increasing order."""formsginout.sorted():print(msg.data)defcollatz(q:ezq.Q,out:ezq.Q)->None:"""Read numbers and compute values."""formsginq:num=float(msg.data)ifmsg.kind=="EVEN":out.put((num,num/2),order=msg.order)elifmsg.kind=="ODD":out.put((num,3*num+1),order=msg.order)defmain()->None:"""Run several threads with a subprocess for printing."""q,out=ezq.Q("thread"),ezq.Q()readers=[ezq.run_thread(collatz,q,out)for_inrange(ezq.NUM_THREADS)]writer=ezq.run(printer,out)fornuminrange(40):kind="EVEN"ifnum%2==0else"ODD"q.put(num,kind=kind,order=num)q.stop(readers)out.stop(writer)if__name__=="__main__":main()LicenseMIT License