Package: concurrent-helper
The Simplest and Most Powerful Concurrent HelperSetuppipinstallconcurrent-helperKey FeaturesSimplest and powerful, very easy to use,only 2 core functions.Works well both onPython2andPython3.Support for multiple concurrent modes:thread pool, process pool and independent multi-processes.Support the mode ofMessage Queue + Service.Multiple progress bar display modes.Quick Startimportconcurrent_helperimportosdefinit(gpu_id):os.environ["CUDA_VISIBLE_DEVICES"]=str(gpu_id)defwork(task_id,gpu_id=None):ifgpu_idisnotNone:os.environ["CUDA_VISIBLE_DEVICES"]=str(gpu_id)print("{}: I am working on{}for{}".format(os.getpid(),os.environ.get("CUDA_VISIBLE_DEVICES"),task_id))returntask_id*2total_gpu_num=3todos=[(x,x%total_gpu_num)forxinrange(10)]rtvs=concurrent_helper.run_with_concurrent(work,todos,"thread")rtvs=concurrent_helper.run_with_concurrent(work,todos,"process")rtvs=concurrent_helper.run_with_concurrent(work,todos,"x-process")rtvs=concurrent_helper.run_with_message_queue(init,list(range(3)),# start 3 serviceswork,list(range(10))# has 10 tasks to do)Core Function: run_with_concurrentdefrun_with_concurrent(func,args_list,concurrent_type="thread",# ["single", "thread", "process", "x-process"]concurrent_num=1,show_process="print",# ["", "tqdm", "print"]show_interval=0.01,):passRun a function by concurrent mode.Key Paramsconcurrent_type:Param ValueDescriptionsinglelike normal for-loopthreadthread poolprocessprocess poolx-processindependent multi-processesWarning:Arrocding to this issue:https://github.com/agronholm/pythonfutures/issues/29, there is a bug inconcurrent.futuresof Python2.
The relevant fix upstream uses Python 3 features and cannot be backported.This bug only happen when child-process killed by system (for exapmle, memory overflow). If you encounter this problem, use thex-processinstead ofprocesswhen you are using Python2.show_process:Param ValueDescription""don't show processtqdmuse tqdm style process barprintprint process bar infoWarning:Please note that tqdm is not thread safe, use print if you need the guarantee of thread safe.show_interval:Param ValueDescription>= 1update progress bar by every N task< 1update progress bar by percentageCore Function: run_with_message_queuedefrun_with_message_queue(init_func,init_args_list,# concurrent_num == len(init_args_list)func,args_list,show_process="print",# ["", "tqdm", "print"]show_interval=0.01,):passRun function byMessage Queue + Servicemode.Fist, start N (N=len(init_args_list)) services, these services will inited byinit_func.After that, these services will obtain M (M=len(args_list)) tasks from message queue and run these byfunc.Why we needMessage Queue + Servicemode?In order to maximize resource utilization (like GPU), we should to start a certain number of services according to the number of resources. Then, these services will obtain tasks from the message queue and run them.Examplesimportconcurrent_helperimportosdefinit(gpu_id):os.environ["CUDA_VISIBLE_DEVICES"]=str(gpu_id)defwork(task_id,gpu_id=None):ifgpu_idisnotNone:os.environ["CUDA_VISIBLE_DEVICES"]=str(gpu_id)print("{}: I am working on{}for{}".format(os.getpid(),os.environ.get("CUDA_VISIBLE_DEVICES"),task_id))returntask_id*2total_gpu_num=3todos=[(x,x%total_gpu_num)forxinrange(5)]rtvs=concurrent_helper.run_with_concurrent(work,todos,"thread",3)print("----")rtvs=concurrent_helper.run_with_concurrent(work,todos,"process",3)print("----")rtvs=concurrent_helper.run_with_concurrent(work,todos,"x-process",3,"tqdm")print("----")rtvs=concurrent_helper.run_with_message_queue(init,list(range(3)),work,list(range(5)))print(rtvs)outputs:37059:Iamworkingon0for037059:Iamworkingon1for137059:Iamworkingon2for2[1/5]......Fnsworkwiththread......in0.0001seconds.37059:Iamworkingon0for337059:Iamworkingon1for4[2/5]......Fnsworkwiththread......in0.0001seconds.[3/5]......Fnsworkwiththread......in0.0003seconds.[4/5]......Fnsworkwiththread......in0.0002seconds.[5/5]......Fnsworkwiththread......in0.0001seconds.
>>>>>>Fns5workwiththreadtotaluse0.0020seconds.
----37063:Iamworkingon0for037064:Iamworkingon1for137064:Iamworkingon0for337065:Iamworkingon2for237063:Iamworkingon1for4[1/5]......Fnsworkwithprocess......in0.0003seconds.[2/5]......Fnsworkwithprocess......in0.0003seconds.[3/5]......Fnsworkwithprocess......in0.0000seconds.[4/5]......Fnsworkwithprocess......in0.0000seconds.[5/5]......Fnsworkwithprocess......in0.0006seconds.
>>>>>>Fns5workwithprocesstotaluse0.0126seconds.
----37066:Iamworkingon0for037067:Iamworkingon1for137068:Iamworkingon2for237069:Iamworkingon0for337070:Iamworkingon1for4[work/x-process]:100%|█████████████████|5/5[00:00<00:00,346.26it/s]----37074:Iamworkingon0for037075:Iamworkingon1for137076:Iamworkingon2for2[1/1]......Fnsworkwithx-process......in0.0003seconds.
>>>>>>Fns1workwithx-processtotaluse0.0085seconds.[1/1]......Fnsworkwithx-process......in0.0004seconds.[1/5]......Fnsworkwithrun_with_message_queue......in0.0090seconds.
>>>>>>Fns1workwithx-processtotaluse0.0090seconds.[1/1]......Fnsworkwithx-process......in0.0003seconds.
>>>>>>Fns1workwithx-processtotaluse0.0087seconds.[2/5]......Fnsworkwithrun_with_message_queue......in0.0093seconds.[3/5]......Fnsworkwithrun_with_message_queue......in0.0090seconds.37077:Iamworkingon0for337078:Iamworkingon1for4[1/1]......Fnsworkwithx-process......in0.0003seconds.
>>>>>>Fns1workwithx-processtotaluse0.0061seconds.[4/5]......Fnsworkwithrun_with_message_queue......in0.0063seconds.[1/1]......Fnsworkwithx-process......in0.0003seconds.
>>>>>>Fns1workwithx-processtotaluse0.0060seconds.[5/5]......Fnsworkwithrun_with_message_queue......in0.0061seconds.
>>>>>>Fns5workwithrun_with_message_queuetotaluse0.0182seconds.[0,2,4,6,8]TODO[DONE] Test codes.[DONE] Detail docs & English describe aboutrun_with_message_queue& More code examples.[DONE] Add paramsshow_process, show_intervaltorun_with_message_queue.[DONE] Removeraise_exceptionparam, it will be default action.