Package: ez-parallel
ez-parallelEasy Parallel MultiprocessingInstallationWithpiporpip3:pipinstall-Uez-parallelorpipinstallez-parallelWithPoetry:poetryaddez-parallelUsageProcess a list of items by using parallel workersDefine what a worker doesDefine how to iterate through the dataJust runDisplay a global progress barDoes the same for multithreadMultithread vs MultiprocessingIn multiprocessing, new processes will be launched, they won't share memory. The user should implement a way
to store the results of a worker and gather these results whenmultiprocess()returns.With multithreading, new threads will be launched, they all share the memory of the parent process. This
also restricts the runtime to a single CPU-core, as threads from a process do not get allocated to different
cores. There will be no performance improvement when the distributed work is CPU-bound.How to choose? (guidelines)CPU-heavy (data transformation, data preprocessing): multiprocessing.IO-heavy (DB requests, File I/O): multithreading.ExamplesHow to process a list?importtimefromez_parallelimportlist_iterator,queue_worker,multiprocess@queue_workerdefwork_one_thing(x:int)->int:# do somethinga=x+2time.sleep(0.1)# Worked on ONE thing = return 1return1# Datathings_to_process=list(range(1000000))# Create the iterator over the things to processiter_fn,nb_things=list_iterator(things_to_process)# Process all the things in parallel with 20 processesmultiprocess(worker_fn=work_one_thing,input_iterator_fn=iter_fn,total=nb_things,nb_workers=20,description='Process the things')How to Process a list by batch?importtimefromtypingimportListfromez_parallelimportbatch_iterator_from_sliceable,queue_worker,multiprocess@queue_workerdefwork_one_thing(x:List[int])->int:# do somethinga=[y+2foryinx]time.sleep(0.1)# Worked on ONE thing = return 1returnlen(x)# Datathings_to_process=list(range(1000000))# Create the iterator over the things to process# This will yield batches of 128 thingsiter_fn=batch_iterator_from_sliceable(items=things_to_process,batch_size=128)nb_things=len(things_to_process)# Process all the things in parallel with 20 processesmultiprocess(worker_fn=work_one_thing,input_iterator_fn=iter_fn,total=nb_things,nb_workers=20,description='Process the things')How to Process a list by batch in multithread?importtimefromtypingimportListfromez_parallelimportbatch_iterator_from_sliceable,queue_worker,multithread@queue_workerdefwork_one_thing(x:List[int])->int:# do somethinga=[y+2foryinx]time.sleep(0.1)# Worked on ONE thing = return 1returnlen(x)# Datathings_to_process=list(range(1000000))# Create the iterator over the things to process# This will yield batches of 128 thingsiter_fn=batch_iterator_from_sliceable(items=things_to_process,batch_size=128)nb_things=len(things_to_process)# Process all the things in parallel with 20 processesmultithread(worker_fn=work_one_thing,input_iterator_fn=iter_fn,total=nb_things,nb_workers=20,description='Process the things')How to collect results in multiprocessing?(Suggestion using temporary files)
In this scenario, results are recorded as JSONL files, the final result is the concatenation of all files.importglobimportjsonimportosimportrandomimportshutilimportstringimporttempfilefromtypingimportListfromez_parallelimportbatch_iterator_from_sliceable,queue_worker,multithreaddefrandom_file_name()->str:return''.join(random.choices(string.ascii_letters,k=32))# All processes write in the same file# The OS will deal with concurrent accesstmp_file=os.path.join(tempfile.gettempdir(),random_file_name())@queue_workerdefwork_one_thing(x:List[int])->int:# This call is blocking until the file can be writtenwithopen(tmp_file,'a')asout:fornumberinx:out.write(json.dumps({"number":number,"square":number**2})+'\n')# Worked on ONE thing = return 1returnlen(x)# Datathings_to_process=list(range(1000000))# Create the iterator over the things to process# This will yield batches of 128 thingsiter_fn=batch_iterator_from_sliceable(items=things_to_process,batch_size=128)nb_things=len(things_to_process)# Process all the things in parallel with 20 processesmultithread(worker_fn=work_one_thing,input_iterator_fn=iter_fn,total=nb_things,nb_workers=20,description='Process the things')# Collect all the datawithopen(tmp_file,'r')assrc:data=[json.loads(line)forlineinsrc]# Delete temporary fileos.remove(tmp_file)How to collect results in multithreadingA lot easier and straightforward, because all the threads share the same memory.fromtypingimportListfromez_parallelimportbatch_iterator,queue_worker,multithread# List are threadsafe in Pythonresults=[]@queue_workerdefwork_one_thing(x:List[int])->int:# do somethingresults.extend({"number":y,"square":y**2}foryinx)# Worked on ONE thing = return 1returnlen(x)# Datathings_to_process=list(range(1000000))# Create the iterator over the things to process# This will yield batches of 128 thingsiter_fn,nb_things=batch_iterator(items=things_to_process,batch_size=128)# Process all the things in parallel with 20 processesmultithread(worker_fn=work_one_thing,input_iterator_fn=iter_fn,total=nb_things,nb_workers=20,description='Process the things')print(len(results))🛡 LicenseThis project is licensed under the terms of theMITlicense. SeeLICENSEfor more details.📃 Citation@misc{ez-parallel,
  author = {Julien Rossi},
  title = {Easy Parallel Multiprocessing},
  year = {2021},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/j-rossi-nl/ez-parallel}}
}CreditsThis project was generated withpython-package-template.