Package: gitermap
gitermap: Easy parallelizable and cacheable list comprehensionsList comprehensions andmap()operations are great in Python, but sometimes it would be nice if they justdid more. gitermap allows users to work through a map operation with seemlessly integrated parallelization and automatic end-caching or step-by-step caching within your workflow. Key functionalities include:Easy parallelization built on top ofjoblibAutomatic caching of results at the end of iteration (end-caching)Automatic caching at each step of iteration (chunk-caching)Additive caching for randomized repeatsSeemless integration withitertools, allowing for generators and functional computinggitermapprovides simple access with useful accessory functions, and detailed access through
exposure to underlying classes viaMapContextwhich neatly wrap the complexity for you.ExamplesSee below for an example that is identical tomap()function, except it returns a list instead of an iterable:>>>fromgitermapimportumap>>>umap(lambdax:x**2,[1,3,5])[1,9,25]This example works exactly asmap()would do, except that with thetqdmpackage installed, a progressbar will also display, which is incredibly handy if each iteration off(x)takes a longish time. But we take this even further; for long runs saving the result at the end is particularly handy to prevent the temptation of re-runs. We follow the convention of adding appropriate characters to the end of function names depending on need:>>># umap with end-caching>>>fromgitermapimportumapc>>>umapc("temp.pkl",lambdax,y:x**2+y**2,[1,3,5],[2,4,6])[5,25,61]Under the hood,umapcuses joblib to dump the data to "temp.pkl" which is in the local directory of wherever the python code is running. The only requirement is that
the function f(x) must return something picklable for joblib to use. Assuming independence between samples, we can perform parallelism across the iterable list usingumapp:>>># umap with parallelism>>>fromgitermapimportumapp>>># creates three threads>>>umapp(lambdax:x**2,[1,3,5])[1,9,25]For particularly long runs, it may be necessary to store the result at each iteration rather than just at the end. This means that if a sneaky bug appears in one of your iterations, all of the computed data can be read in up to the point of the bug, meaning your compute pipeline doesn't need to be fully re-computed:>>># umap with caching by chunks>>>fromgitermapimportumapcc>>># no threading, saving each iteration in a subfile>>>umapcc("temp.pkl",lambdax:x**2,range(50))[1,9,...]Note that at the end ofumapcc, the temporary directory and files are deleted, leaving only "temp.pkl". See the below table summary of each function and what functionality it supports:Function NameParallelizationEnd-CachingChunk-Cachingumap☒☒☒umapp☑☒☒umapc☒☑☒umappc☑☑☒umapcc☒☑☑umappcc☑☑☑For more control over how many threads are used for parallelization, end sounds, keyword arguments inf(...)and so on, we expose an object calledMapContext, passing in then_jobsargument as you would forjoblibor in scikit-learn (default=None, or 1):>>>importitertoolsasit>>>fromgitermapimportMapContext>>>withMapContext(n_jobs=-1)asctx:>>>result=ctx.compute(lambdax:x**2,it.islice(it.count(),0,100))>>>result[0,1,4,9,...]Note that this is equivalent toumap, albeit in longer form - under the hoodumapsimply creates
aMapContextobject and calls compute like so. Note that if you wish to have lazy evaluation by deferring execution,
we provide this through thereturn_typeparameter:>>>fromgitermapimportMapContext>>>withMapContext(return_type="generator")asctx:>>>result=ctx.compute(lambdax:x**2,range(20))>>>result<generatorobjectMapContext._map_comp.<locals>.<genxpr>at0x00000000000>Note that generators do not also perform parallelization when evoked.
For further details, see the example notebooks within this project.RequirementsThe following requirements are essential forgitermap:Python >= 3.8joblib>= 1.4The following packages are highly recommended but not essential:tqdm>= 1.0: For progressbarsFor ending sound inMapContext, you will also need:numpy: For processing sound wavessimpleaudio: For playing soundsThe other packages such asitertools,functoolsand more are default and come with the standard Python distribution.