Package: parhugin
Parallelize Python codesparhuginprovides functions to:run one or more Python functions in parallel using multiprocessingTable of contentsInstallation and setupRun one or more Python functions in parallel using multiprocessingComplete examplesInstallationinstall using pippipinstallgit+https://github.com/kasra-hosseini/parhugin.gitinstall parhugin from the source code:Clone parhugin source code:gitclonehttps://github.com/kasra-hosseini/parhugin.gitInstall parhugin:cd /path/to/my/parhugin
python setup.py installAlternatively:cd /path/to/my/parhugin
pip install -v -e .Run one or more Python functions in parallel using multiprocessingIn this scenario, we have:one or more functionsa list of jobs to be run in parallel, e.g.:[[func1,(arg1_1,arg2_1,arg3_1)],[func1,(arg1_2,arg2_2,arg3_2)],[func2,(...)],...]⚠️ If a function has only one argument, do not forget to add it to the above list either[func_one_arg, [arg1]]or[func_one_arg, (arg1,)].User specifies the number of processes to be run in parallel.parhuginparallelizes by distributing the jobs following FIFO on the requested number of processes.Example 1First, we importparhuginand define two simple functions calledfunc1andfunc2. These functions can have different number of arguments.fromparhuginimportmultiFuncimporttime# Define two simple functions, func1 and func2# Note that functions can have different number of argumentsdeffunc1(a,b,sleep=0.5,info="func1"):print(f"start,{info}calculated{a+b}")time.sleep(sleep)print(f"end,{info}")deffunc2(a,sleep=0.2,info="func2"):print(f"start,{info}prints{a}")time.sleep(sleep)print(f"end,{info}")Next, we specify the number of processes to run in parallel. This can be the number of processors if the jobs are CPU-intensive. Otherwise, you can set this to any other values.myproc=multiFunc(num_req_p=10)Now, we need to add jobs to be run in parallel. There are different ways to do this:Add one function and its arguments:myproc.add_job(target_func=func1,target_args=(2,3,0.5,"func1"))print(myproc)Similarly, we can add another function:myproc.add_job(target_func=func2,target_args=(10,0.2,"func2"))print(myproc)Create a list of jobs:list_jobs=[]foriinrange(1,20):list_jobs.append([func2,(f"{i}",0.2,"func2")])# and then adding them to myprocmyproc.add_list_jobs(list_jobs)print(myproc)Finally, run the jobs on the requested number of processes:myproc.run_jobs()It is also possible to change the verbosity level of the output by:myproc.run_jobs(verbosity=2)Complete examplesExample 1fromparhuginimportmultiFuncimporttime# Define two simple functions, func1 and func2# Note that functions can have different number of argumentsdeffunc1(a,b,sleep=0.5,info="func1"):print(f"start,{info}calculated{a+b}")time.sleep(sleep)print(f"end,{info}")deffunc2(a,sleep=0.2,info="func2"):print(f"start,{info}prints{a}")time.sleep(sleep)print(f"end,{info}")myproc=multiFunc(num_req_p=10)myproc.add_job(target_func=func1,target_args=(2,3,0.5,"func1"))print(myproc)myproc.add_job(target_func=func2,target_args=(10,0.2,"func2"))print(myproc)list_jobs=[]foriinrange(1,20):list_jobs.append([func2,(f"{i}",0.2,"func2")])# and then adding them to myprocmyproc.add_list_jobs(list_jobs)print(myproc)myproc.run_jobs(verbosity=2)