Package: parallelbar
ParallelbarTable of contentsInstalationUsageException handlingChangelogNew in version 2.4New in version 2.3New in version 1.3New in version 1.2New in version 1.1New in version 1.0New in version 0.3Problems of the naive approachLicenseParallelbardisplays the progress of tasks in the process pool forPoolclass methods such asmap,starmap(since 1.2 version),imapandimap_unordered. Parallelbar is based on thetqdmmodule and the standard pythonmultiprocessinglibrary.
Also, it is possible to handle exceptions that occur within a separate process, as well as set a timeout for the execution of a task by a process.Installationpipinstallparallelbarorpipinstall--usergit+https://github.com/dubovikmaster/parallelbar.gitUsagefromparallelbarimportprogress_imap,progress_map,progress_imapufromparallelbar.toolsimportcpu_bench,fibonacciLet's create a list of 100 numbers and testprogress_mapwith default parameters on a toy functioncpu_bench:tasks=range(10000)%%timelist(map(cpu_bench,tasks))Walltime:52.6sOk, by default this works on one core of my i7-9700F and it took 52 seconds. Let's parallelize the calculations for all 8 cores and look at the progress. This can be easily done by replacing standart functionmapwithprogress_map.if__name__=='__main__':progress_map(cpu_bench,tasks)Core progress:You can also easily useprogress_imapandprogress_imapuanalogs of theimapandimap_unorderedmethods of thePool()class%%timeif__name__=='__main__':tasks=[20+iforiinrange(15)]result=progress_imap(fibonacci,tasks,chunk_size=1,core_progress=False)Exception handlingYou can handle exceptions and set timeouts for the execution of tasks by the process.Consider the following toy example:deffoo(n):ifn==5orn==17:1/0elifn==10:time.sleep(2)else:time.sleep(1)returnnif__name__=='__main__':res=progress_map(foo,range(20),process_timeout=5,n_cpu=8)As you can see, under the main progress bar, another progress bar has appeared that displays the number of tasks that ended unsuccessfully. At the same time, the main bar turned orange, as if signaling something went wrongprint(res)[0,1,2,3,4,ZeroDivisionError('division by zero'),6,7,8,9,10,11,12,13,14,15,16,ZeroDivisionError('division by zero'),18,19]In the resulting array, we have exceptions in the corresponding places. Also, we can see the exception traceback:print(res[5].traceback)Traceback(mostrecentcalllast):File"/home/padu/anaconda3/envs/work/lib/python3.9/site-packages/pebble/common.py",line174,inprocess_executereturnfunction(*args,**kwargs)File"/home/padu/anaconda3/envs/work/lib/python3.9/site-packages/parallelbar/parallelbar.py",line48,in_processresult=func(task)File"/tmp/ipykernel_70395/285585760.py",line3,infoo1/0ZeroDivisionError:divisionbyzeroFrom which concept at what place in the code the exception occurred.
Let's add a timeout of 1.5 seconds for each process. If the process execution time exceeds 1.5 seconds, an appropriate exception will be raised and handled. In this case, the process will restart and continue to work (thanks topebble)if__name__=='__main__':res=progress_map(foo,range(20),process_timeout=1.5,n_cpu=8)print(res)[0,1,2,3,4,ZeroDivisionError('division by zero'),6,7,8,9,'function foo took longer than 1.5 s.',11,12,13,14,15,16,ZeroDivisionError('division by zero'),18,19]Exception handling has also been added to methodsprogress_imapandprogress_imapu.ChangelogVersion 2.4fixedissueForWindows OS, when using theadd_progressdecorator, the function being decorated no longer needs theworker_queuekeyword argument.New in version 2.3addedwrappersmodule with which contains decorators:stop_it_after_timeout- stops the function execution after the specified time (in seconds)add_progress- adds a progress bar to the function execution, exception handling and timeout.Usage example for UNIX systems:fromparallelbar.wrappersimportadd_progressfromparallelbarimportprogress_mapimporttime@add_progress(error_handling='coerce',timeout=.5)deffoo(n):ifn==5orn==17:1/0elifn==10:time.sleep(1)else:time.sleep(.1)returnndefbar(x):return[foo(i)foriinrange(x)]if__name__=='__main__':# you must specify the total number of tasksres=progress_map(bar,[10,20,30,40],n_cpu=4,total=100)Out:ForWindowssystems you need to add theworker_queueparameter to the functionsfooandbarand use theused_add_progress_decoratorparameter in theprogress_mapfunction:@add_progress(error_handling='coerce',timeout=.5)deffoo(n):ifn==5orn==17:1/0elifn==10:time.sleep(1)else:time.sleep(.1)returnndefbar(x,worker_queue=None):return[foo(i,worker_queue=worker_queue)foriinrange(x)]if__name__=='__main__':res=progress_map(bar,[10,20,30,40],n_cpu=4,total=100,used_add_progress_decorator=True)Out:You can also use thestopit_after_timeoutdecorator separately:fromparallelbar.wrappersimportstopit_after_timeoutfromparallelbarimportprogress_mapimporttime@stopit_after_timeout(.5,raise_exception=True)deffoo(n):ifn==5:time.sleep(1)else:time.sleep(.1)returnnif__name__=='__main__':print(f'first result is:{foo(3)}')print(f'second result is:{foo(5)}')Out:firstresultis:3TimeoutErrorTraceback(mostrecentcalllast)CellIn[7],line1614if__name__=='__main__':15print(foo(3))--->16print(foo(5))File/opt/conda/envs/user_response/lib/python3.10/site-packages/parallelbar/wrappers.py:38,instopit_after_timeout.<locals>.actual_decorator.<locals>.wrapper(*args,**kwargs)36msg=f'function took longer than{s}s.'37ifraise_exception:--->38raiseTimeoutError(msg)39result=msg40finally:TimeoutError:functiontooklongerthan0.5s.addedreturn_failed_taskskeyword parameter to theprogress_map/starmap/imap/imapufunction (default=False) - ifTruethen the result will include the tasks that failed with an exception.New in version 1.3addedmaxtaskperchildkeyword parameter to theprogress_map/starmap/imap/imapufunction (default=None)New in version 1.2Addedprogress_starmapfunction. An extension of thestarmapmethod of thePoolclass.Improved documentation.New in version 1.1Thebar_stepkeyword argument is no longer used and will be removed in a future versionAddedneed_serializeboolean keyword argument to theprogress_map/imap/imapufunction (defaultFalse). Requiresdillto be installed. IfTruethe target function is serialized usingdilllibrary. Thus, as a target function, you can now use lambda functions, class methods and other callable objects thatpicklecannot serializeAdded dynamic optimization of the progress bar refresh rate. This can significantly improve the performance of theprogress_map/imap/imapufunctions ror very long iterables and small execution time of one task by the objective function.New in version 1.0The "ignore" value of theerror_behaviorkey parameter is no longer supported.Default value of key parametererror_behaviorchanged to "raise".Thepebblemodule is no longer used.Added key parameterexecutorin the functionsprogress_map,progress_imapandprogress_imapu. Must be one of the values:"threads" - use thread pool"processes" - use processes pool (default)New in version 0.3.0Theerror_behaviorkeyword argument has been added to theprogress_map,progress_imapandprogress_imapumethods.
Must be one of the values: "raise", "ignore", "coerce"."raise" - raise an exception thrown in the process pool."ignore" - ignore the exceptions that occur. Do not add anything to the result"coerce" - handle the exception. The result will include the value set by the parameterset_error_value(by default None - the traceback of the raised exception will be added to the result)Theset_error_valuekeyword argument has been added to theprogress_map,progress_imapandprogress_imapumethods.Example of usageimporttimeimportresourceasrsfromparallelbarimportprogress_imapdefmemory_limit(limit):soft,hard=rs.getrlimit(rs.RLIMIT_AS)rs.setrlimit(rs.RLIMIT_AS,(limit,hard))defmy_awesome_foo(n):ifn==0:s='a'*10000000elifn==20:time.sleep(100)else:time.sleep(1)returnnif__name__=='__main__':tasks=range(30)start=time.monotonic()result=progress_imap(my_awesome_foo,tasks,process_timeout=1.5,initializer=memory_limit,initargs=(100,),n_cpu=4,error_behavior='coerce',set_error_value=None,)print(f'time took:{time.monotonic()-start:.1f}')print(result)time took: 8.2
[MemoryError(), 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 
16, 17, 18, 19, TimeoutError('function "my_awesome_foo" took longer than 1.5 s.'), 21, 22, 23, 24, 25, 26, 27, 28, 29]Set NaN instead of tracebacks to the result of the pool operation:if__name__=='__main__':tasks=range(30)start=time.monotonic()result=progress_imap(my_awesome_foo,tasks,process_timeout=1.5,initializer=memory_limit,initargs=(100,),n_cpu=4,error_behavior='coerce',set_error_value=float('nan'),)print(f'time took:{time.monotonic()-start:.1f}')print(result)time took: 8.0
[nan, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 
16, 17, 18, 19, nan, 21, 22, 23, 24, 25, 26, 27, 28, 29]Let's ignore exception:if__name__=='__main__':tasks=range(30)start=time.monotonic()result=progress_imap(my_awesome_foo,tasks,process_timeout=1.5,initializer=memory_limit,initargs=(100,),n_cpu=4,error_behavior='ignore',set_error_value=None,)print(f'time took:{time.monotonic()-start:.1f}')print(result)time took: 8.0
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 
16, 17, 18, 19, 21, 22, 23, 24, 25, 26, 27, 28, 29]Problems of the naive approachWhy can't I do something simpler? Let's take the standardimapmethod and run through it in a loop withtqdmand take the results from the processes:frommultiprocessingimportPoolfromtqdm.autoimporttqdmif__name__=='__main__':withPool()asp:tasks=[20+iforiinrange(15)]pool=p.imap(fibonacci,tasks)result=[]foriintqdm(pool,total=len(tasks)):result.append(i)It looks good, doesn't it? But let's do the following, make the first task very difficult for the core. To do this, I will insert the number 38 at the beginning of the tasks list. Let's see what happensif__name__=='__main__':withPool()asp:tasks=[20+iforiinrange(15)]tasks.insert(0,39)pool=p.imap_unordered(fibonacci,tasks)result=[]foriintqdm(pool,total=len(tasks)):result.append(i)This is a fiasco. Our progress hung on the completion of the first task and then at the end showed 100% progress.
Let's try to do the same experiment only for the progress_imap function:if__name__=='__main__':tasks=[20+iforiinrange(15)]tasks.insert(0,39)result=progress_imap(fibonacci,tasks)The progress_imap function takes care of collecting the result and closing the process pool for you.
In fact, the naive approach described above will work for the standard imap_unordered method. But it does not guarantee the order of the returned result. This is often critically important.LicenseMIT license