Package: blazer
An HPC abstraction over MPI that uses pipes and pydash primitives. Blazer will handle all the MPI orchestration behind the scenes for you. You just work strictly with data and functions. Easy!InstallFrom pypi$pipinstallblazerFrom clone$gitclonehttps://github.com/radiantone/blazer
$cdblazer
$makeinitinstallNOTE: For some tests ensure you have slurm configured properly (single or muli-machine). However, using slurm is not required to use blazer.Linting$makelintTests(venv)$mpirun-n2pythonsetup.pytestblazer/tests/test_parallel.py::test_parallelPASSED[50%]blazer/tests/test_pipeline.py::test_pipelinePASSED[100%]==============================2passedin0.48s===============================ctrl-cor$./bin/tests.shExamplesimportblazerfromblazer.hpc.mpiimportparallel,pipeline,partialasp,scatter,where,select,filter,rank,sizedefcalc_some(value,*args):""" Do some calculations """result={'some':value}returnresultdefcalc_stuff(value,*args):""" Do some calculations """result={'this':value}returnresultdefadd_date(result):fromdatetimeimportdatetimeiftype(result)isdict:result['date']=str(datetime.now())returnresultdefcalc_more_stuff(result):""" Do some more calculations """iftype(result)islist:result+=[{'more':'stuff'}]eliftype(result)isdict:result['more']='stuff'returnresultINPUT_DATA='that'withblazer.begin():result1=parallel([p(calc_stuff,1),p(calc_stuff,2),p(calc_stuff,3),p(calc_stuff,4),p(calc_stuff,5)])blazer.print("PARALLEL1:",result1)ifblazer.ROOT:r=list(result1|where(lambdag:where(lambdag:g['this']>1))|select(lambdag:p(calc_stuff,g['this']*2)))# Run the composed computation in parallel, wait for resultresult=parallel(r)blazer.print("PARALLEL2:",result)r=pipeline([p(calc_stuff,'DATA'),p(pipeline,[calc_some,add_date]),calc_stuff])blazer.print("PIPELINE:",r)scatter_data=scatter(list(range(0,(size*2)+2)),calc_some)blazer.print("SCATTER_DATA:",scatter_data)result=pipeline([p(calc_stuff,INPUT_DATA),add_date,scatter_data,p(parallel,[calc_some,p(pipeline,[calc_stuff,p(parallel,[calc_some,calc_some]),calc_stuff]),calc_some]),calc_more_stuff])blazer.print("PIPELINE RESULT:",result)defget_data():""" Data generator """foriinrange(0,(size*2)):yieldiresult=scatter(get_data(),calc_some)blazer.print("SCATTER:",result)To run:(venv) $ export PYTHONPATH=.
(venv) $ mpirun -n 4 python blazer/examples/example1.py 
PARALLEL1: [{'this': 1}, {'this': 2}, {'this': 3}, {'this': 4}, {'this': 5}]
PARALLEL2: [{'this': 4}, {'this': 6}, {'this': 2}, {'this': 8}, {'this': 10}]
PIPELINE: {'this': {'some': ({'this': 'DATA'},), 'date': '2022-02-11 02:47:23.356461'}}
SCATTER_DATA: [{'some': 0}, {'some': 1}, {'some': 2}, {'some': 3}, {'some': 4}, {'some': 5}, {'some': 6}, {'some': 7}, {'some': 8}, {'some': 9}, {'some': None}, {'some': None}]
PIPELINE RESULT: [{'this': [{'this': ([{'some': 0}, {'some': 1}, {'some': 2}, {'some': 3}, {'some': 4}, {'some': 5}, {'some': 6}, {'some': 7}, {'some': 8}, {'some': 9}, {'some': None}, {'some': None}],)}, {'some': {'some': [{'some': 0}, {'some': 1}, {'some': 2}, {'some': 3}, {'some': 4}, {'some': 5}, {'some': 6}, {'some': 7}, {'some': 8}, {'some': 9}, {'some': None}, {'some': None}]}}]}, {'some': 'some'}, {'more': 'stuff'}]
[0, 1, 2, 3, 4, 5, 6, 7]
SCATTER: [{'some': 0}, {'some': 1}, {'some': 2}, {'some': 3}, {'some': 4}, {'some': 5}, {'some': 6}, {'some': 7}]A map/reduce exampleimportblazerfromblazer.hpc.mpiimportmap,reducedefsqr(x):returnx*xdefadd(x,y=0):returnx+ywithblazer.begin():result=map(sqr,[1,2,3,4])blazer.print(result)result=reduce(add,result)blazer.print(result)To run:(venv) $ export PYTHONPATH=.
(venv) $ mpirun -n 4 python blazer/examples/example3.py 
[1, 4, 9, 16]
30Streaming ComputeBlazer supports the notion ofstreaming computeto handle jobs where the data can't fit into memory on a single machine.
In this example we implement a map/reduce computation where everything is streaming from the source data through the results without loading all the data into memory.""" Streaming map/reduce example """fromitertoolsimportgroupbyfromrandomimportrandrangefromtypingimportGeneratorimportblazerfromblazer.hpc.mpiimportstreamdefdatagen()->Generator:foriinrange(0,1000):r=randrange(2)v=randrange(100)ifr:yield{"one":1,"value":v}else:yield{"zero":0,"value":v}defkey_func(k):returnk["key"]defmap(datum):datum["key"]=list(datum.keys())[0]returndatumdefreduce(datalist):fromblazer.hpc.mpiimportrank_list=sorted(datalist,key=key_func)grouped=groupby(_list,key_func)return[{"rank":rank,key:list(group)}forkey,groupingrouped]withblazer.begin():importjsonmapper=stream(datagen(),map,results=True)reducer=stream(mapper,reduce,results=True)ifblazer.ROOT:forresultinreducer:blazer.print("RESULT",json.dumps(result,indent=4))NOTE: blazer has (currently) only been tested onmpirun (Open MPI) 4.1.0OverviewBlazer is ahigh-performance computing(HPC) library that hides the complexities of a super computer'smessage-passing interfaceor (MPI).
Users want to focus on their code and their data and not fuss with low-level API's for orchestrating results, building pipelines and running fast, parallel code. This is why blazer exists!With blazer, a user only needs to work with simple, straightforward python. No cumbersome API's, idioms, or decorators are needed.
This means they can get started quicker, run faster code, and get their jobs donefaster!General DesignBlazer is designed around the concept of computingprimitivesor operations. Some of the primitives include:parallel- For computing a list of tasks in parallelpipeline- For computing a list of tasks in sequence, passing the results alongmap- For mapping a task to a datasetreduce- For mapping a task to a data list and computing a single resultIn addition there are other primitives to help manipulate lists of tasks or data, such as:where- Filter a list of tasks or data elements based on a function or lambdaselect- Apply a function to each list element and return the resultContext HandlersBlazer uses convenient context handlers to control blocks of code that need to be scheduled to MPI processes behind the scenes.
There are two types of context handlers currently.MPI Context Handlerblazer.begin()is a mandatory context that enables the MPI scheduler behind the various primitives to operate correctly.importblazerblazer.begin():defget_data():""" Data generator """foriinrange(0,(size*2)):yieldiresult=scatter(get_data(),calc_some)blazer.print("SCATTER:",result)GPU Context Handlerblazer.gpu()is a context that requests (from the invisible MPI scheduler) dedicated access to a specific GPU on your MPI node fabric.importloggingimportblazerimportnumpyasnpfromblazer.hpc.mpi.primitivesimporthost,rankfromnumbaimportvectorizefromtimeitimportdefault_timerastimerdefdovectors():@vectorize(['float32(float32, float32)'],target='cuda')defdopow(a,b):returna**bvec_size=100a=b=np.array(np.random.sample(vec_size),dtype=np.float32)c=np.zeros(vec_size,dtype=np.float32)start=timer()dopow(a,b)duration=timer()-startreturndurationwithblazer.begin(gpu=True):# on-fabric MPI schedulerwithblazer.gpu()asgpu:# on-metal GPU scheduler# gpu object contains metadata about the GPU assignedprint(dovectors())Shared Data Context HandlerBlazer supports synchronizing shared data across ranks seamlessly. Here is an example of sharing a dictionary where each rank adds its own data to the dictionary and it is available to all other ranks magically!fromrandomimportrandrangeimportblazerfromblazer.hpc.mpi.primitivesimportrankwithblazer.environment()asvars:rv=randrange(10)vars["rank"+str(rank)]=[{"key":randrange(10)},randrange(10),randrange(10),randrange(10),]print("RANK:",rank,"DATA",vars.vars)blazer.print(vars["rank1"])Cross-Cluster SupercomputingBlazer comes with a built-in design pattern for performing cross-cluster HPC. This is useful if you want to allocate compute resources on different super-computers and then build a pipeline of jobs across them. Here is a simple example using ALCF's Cooley and Theta systems (which are built into blazer).fromblazer.hpc.alcfimportcooley,thetagpufromblazer.hpc.localimportparallel,pipeline,partialasp# Log into each cluster using MFA password from MobilePASScooleyjob=cooley.job(user='dgovoni',n=1,q="debug",A="datascience",password=True,script="/home/dgovoni/git/blazer/testcooley.sh").login()thetajob=thetagpu.job(user='dgovoni',n=1,q="single-gpu",A="datascience",password=True,script="/home/dgovoni/git/blazer/testthetagpu.sh").login()defhello(data,*args):return"Hello "+str(data)# Mix and match cluster compute jobs with local code tasks# in serial chainingcooleyjob("some data").then(hello).then(thetajob).then(hello)# Run a cross cluster compute jobresult=pipeline([p(thetajob,"some data2"),p(cooleyjob,"some data1")])print("Done")When each job.login()method is run, it will gather the MFA login credentials for that system and then use that to schedule jobs on that system via ssh.Notice the use of thepipelineprimitive above. It's the same primitive you would use to build your compute workflows! Composable tasks and composable super-computers.