Package: pbatch
pbatchParallel batch processing on top of regular python functionsInstallationRequires python 3.7+pipinstallpbatchUsagepbatch.pmapSimilar to built-inmap, but executes the function in
parallel. Number of concurrent executions can be limited through achunk_sizekeyword argument.importtimeimportpbatchdeflong_square(x):time.sleep(1)print(x)returnx**2list(map(long_square,[1,2,3]))# 1# 2# 3# => [1, 4, 9] (after 3 seconds)list(pbatch.pmap(long_square,[1,2,3]))# 1# 2# 3# => [1, 4, 9] (after 1 second)list(pbatch.pmap(long_square,[1,2,3],chunk_size=2))# 1# 2# 3# => [1, 4, 9] (after 2 seconds)Supports multiple-arity functions exactly asmapdoes:importtimeimportpbatchdefmultiple_args(a,b,c):print(f"a={a}, b={b}, c={c})time.sleep(1)returnclist(map(multiple_args,[1,2],[60,70],[1000,2000]))# a=1, b=60, c=1000# a=2, b=70, c=2000# => [1000, 2000] (after 2 seconds)list(pbatch.pmap(multiple_args,[1,2],[60,70],[1000,2000]))# a=1, b=60, c=1000# a=2, b=70, c=2000# => [1000, 2000] (after 1 second)list(pbatch.pmap(multiple_args,[1,2],[60,70],[1000,2000],chunk_size=1))# a=1, b=60, c=1000# a=2, b=70, c=2000# => [1000, 2000] (after 2 second)Note that if one iterable is shorter than the rest, remaining elements
in the other iterators will be ignored.If an exception is raised when processing an item, the remaining
elements in the current chunk will be completed and then apbatch.PMapExceptionwill be raised, including the results and
exception from thecurrentchunk.If any of the subtasks raises an exception, apbatch.PMapExceptionwill be raised:defraise_on_two(x):ifx==2:raiseValueError("Number is two")returnxtry:list(pbatch.pmap(raise_on_two,[1,2,3]))exceptpbatch.PMapExceptionase:e.results# => [1, ValueError("Number is two"), 3]e.exceptions# => [ValueError("Number is two")]str(e)# => "[1, ValueError('Number is two'), 3]"repr(e)# => "[1, ValueError('Number is two'), 3]"If directly converting the results to a list, as above, and an
exception is raised after the first chunk successfully completes, the
results from the first chunk will be forgotten. If such results are
important, it is better to manually process each item out of the
generator, as chunks are generated:results=[]try:forresultinpbatch.pmap(...):results.append(result)exceptpbatch.PMapExceptionase:results.extend(e.results)After executing,resultswill contain all results that were
successfully processed without raising an exception.Alternatively, wrap the function being mapped in a try/except block to
have more full control over when aPMapExceptionwill be raised.pbatch.postponeBegin execution of a function without blocking code execution (until.wait()is called)importtimeimportpbatchdeflong_function(x,power=2):time.sleep(1)returnx**powerpostponement=pbatch.postpone(long_function,3,power=3)time.sleep(1)result=postponement.wait()# does not wait 1 second anymorepbatch.partitionSplit up an iterable into fixed-sized chunks (except the final chunk
in some cases)Returns a generator that yields lists of elements (chunks)importpbatchpartitions=list(pbatch.partition([1,2,3,4,5,6,7,8,9,10],chunk_size=4))# => [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10]]Chunks are lazily generated:defprint_return(x):print(x)returnxnext(pbatch.partition(map(print_return,range(10)),4))# 0# 1# 2# 3# => [0, 1, 2, 3]DevelopmentClone the repo, then from the project directory:python3.7-mvenv.venv
..venv/bin/activate

makeinstall-devTo run tests (and show coverage):maketestBefore making changes, fix formatting and check changes (isort, black, flake8, mypy:makeformatcheck0.2.0 2020-09-22(not backwards compatible)makepmapreturn a generator, so that the entire result set does
not necessarily have to be in memory at the same time0.1.1 2020-09-20Remove deprecatedloopargument fromasyncio.wait(fixes warning)0.1.0 2020-09-20Initial release