Package: parameterize-jobs
parameterize_jobsis a lightweight, pure-python toolkit for concisely and clearly creating large, parameterized, mapped job specifications.Free software: MIT licenseDocumentation:https://parameterize-jobs.readthedocs.ioFeaturesExpand a job’s dimensionality by multiplyingComponentSet,Constant, orParallelComponentSetobjectsExtend the number of jobs by addingComponentSet,Constant, orParallelComponentSetobjectsJobs are provided to functions as dictionaries of parametersThe helper decorator@expand_kwargsturns these kwarg dictionaries into
named argument callsWorks seamlessly with many task running frameworks, including dask’sclient.mapand profiling toolsTODOsView and submit issues on theissues page.QuickstartComponentSetobjects are the base objects, and can be defined with any number of named iterables:>>>importparameterize_jobsaspjs>>>a=pjs.ComponentSet(a=range(5))>>>a<ComponentSet{'a':5}>These objects have defined lengths (if the provided iterable has a defined length), and can be indexed and iterated over:>>>len(a)5>>>a[0]{'a':0}>>>list(a)[{'a':0},{'a':1},{'a':2},{'a':3},{'a':4}]Adding twoComponentSetobjects extends the total job length>>>a2=pjs.ComponentSet(a=range(3))>>>a+a2<MultiComponentSet[{'a':5},{'a':3}]>>>>len(a+a2)8>>>list(a+a2)[{'a':0},{'a':1},{'a':2},{'a':3},{'a':4},{'a':0},{'a':1},{'a':2}]Multiplying twoComponentSetobjects expands their dimensionality:>>>b=pjs.ComponentSet(b=range(3))>>>a*b<ComponentSet{'a':5,'b':3}>>>>len(a*b)15>>>(a*b)[-1]{'a':4,'b':2}>>>list(a*b)[{'a':0,'b':0},{'a':0,'b':1},{'a':0,'b':2},{'a':1,'b':0},{'a':1,'b':1},{'a':1,'b':2},{'a':2,'b':0},{'a':2,'b':1},{'a':2,'b':2},{'a':3,'b':0},{'a':3,'b':1},{'a':3,'b':2},{'a':4,'b':0},{'a':4,'b':1},{'a':4,'b':2}]These parameterized job specifications can be used in mappable jobs. The helper decoratorexpand_kwargsmodifies a function to accept a dictionary and expands them into keyword arguments:>>>@pjs.expand_kwargs...defmy_simple_func(a,b,c=1):...returna*b*c>>>list(map(my_simple_func,a*b))[0,0,0,0,0,0,1,2,3,4,0,2,4,6,8,0,3,6,9,12]Jobs do not have to be the combinatorial product of all components:>>>ab1=pjs.ComponentSet(a=[0,1],b=[0,1])>>>ab2=pjs.ComponentSet(a=[10,11],b=[-1,1])>>>list(map(my_simple_func,ab1+ab2))[0,0,0,1,-10,-11,10,11]AConstantobject is simply aComponentSetobject defined with single values passed as keyword arguments rather than iterables passed as keyword arguments:>>>c=pjs.Constant(c=5)>>>list(map(my_simple_func,(ab1+ab2)*c))[0,0,0,5,-50,-55,50,55]AParallelComponentSetobject is simply aMultiComponentSetobject where eachComponentis aConstantobject.>>>pcs=pjs.ParallelComponentSet(a=[1,2],b=[10,20])>>>list(map(my_simple_func,pcs))[10,40]Arbitrarily complex combinations of ComponentSets can be created:>>>c1=pjs.Constant(c=1)>>>c2=pjs.Constant(c=2)>>>list(map(my_simple_func,(ab1+ab2)*c1+(ab1+ab2)*c2))[0,0,0,1,-10,-11,10,11,0,0,0,2,-20,-22,20,22]Anything can be inside aComponentSetiterable, including data, functions, or other objects:>>>transforms=(...pjs.Constant(transform=lambdax:x,transform_name='linear')...+pjs.Constant(transform=lambdax:x**2,transform_name='quadratic'))...>>>fps=pjs.Constant(...read_pattern='source/my-fun-data_{year}.csv',...write_pattern='transformed/my-fun-data_{transform_name}_{year}.csv')>>>years=pjs.ComponentSet(year=range(1980,2018))>>>@pjs.expand_kwargs...defprocess_data(read_pattern,write_pattern,transform,transform_name,year):......df=pd.read_csv(read_pattern.format(year=year))......transformed=transform(df)......transformed.to_csv(...write_pattern.format(...transform_name=transform_name,...year=year))...>>>_=list(map(process_data,transforms*fps*years))This works seamlessly with dask’sclient.mapto provide intuitive job parameterization:>>>importdask.distributedasdd>>>client=dd.LocalClient()>>>futures=client.map(my_simple_func,(ab1+ab2)*c1+(ab1+ab2)*c2)>>>dd.progress(futures)