Package: esi-acme
ACME: Asynchronous Computing Made ESImain:dev:Table of ContentsSummaryInstallationUsageBasic ExamplesIntermediate ExamplesAdvanced UseHandling ResultsLoad Results From FilesCollect Results in Single HDF5 DatasetCollect Results in Local MemoryDebuggingDocumentation and ContactResourcesSummaryThe objective of ACME (pronounced"ak-mee") is to provide easy-to-use
wrappers for calling Python functions concurrently ("embarassingly parallel workloads").
ACME is developed at theErnst Strüngmann Institute (ESI) gGmbH for Neuroscience in Cooperation with Max Planck Societyand released free of charge under theBSD 3-Clause "New" or "Revised" License.
ACME relies heavily on the concurrent processing librarydaskand was primarily designed to facilitate the use ofSLURMon the ESI HPC cluster (although other HPC infrastructure running SLURM can be
leveraged as well). Local multi-processing hardware (i.e., multi-core CPUs)
is fully supported too. ACME is itself used as the parallelization engine ofSyNCoPy.InstallationACME can be installed withpippipinstallesi-acmeor viacondacondainstall-cconda-forgeesi-acmeTo get the latest development version, simply clone our GitHub repository:gitclonehttps://github.com/esi-neuroscience/acme.gitcdacme/
pipinstall-e.UsageBasic ExamplesSimplest use, everything is done automatically.fromacmeimportParallelMapdeff(x,y,z=3):return(x+y)*zwithParallelMap(f,[2,4,6,8],4)aspmap:pmap.compute()See also ourQuickstart Guide.Intermediate ExamplesSet number of function calls vian_inputsimportnumpyasnpfromacmeimportParallelMapdeff(x,y,z=3,w=np.zeros((3,1)),**kwargs):return(sum(x)+y)*z*w.max()pmap=ParallelMap(f,[2,4,6,8],[2,2],z=np.array([1,2]),w=np.ones((8,1)),n_inputs=2)withpmapasp:p.compute()More details inOverride Automatic Input Argument DistributionAdvanced UseAllocate customclientobject and recycle it for several computations
(useslurm_cluster_setupon non-ESI HPC infrastructure orlocal_cluster_setupwhen working on your local machine)importnumpyasnpfromacmeimportParallelMap,esi_cluster_setupdeff(x,y,z=3,w=np.zeros((3,1)),**kwargs):return(sum(x)+y)*z*w.max()defg(x,y,z=3,w=np.zeros((3,1)),**kwargs):return(max(x)+y)*z*w.sum()n_workers=200client=esi_cluster_setup(partition="8GBXS",n_workers=n_workers)x=[2,4,6,8]z=range(n_workers)w=np.ones((8,1))pmap=ParallelMap(f,x,np.random.rand(n_workers),z=z,w=w,n_inputs=n_workers)withpmapasp:p.compute()pmap=ParallelMap(g,x,np.random.rand(n_workers),z=z,w=w,n_inputs=n_workers)withpmapasp:p.compute()For more information seeReuse Worker ClientsHandling ResultsLoad Results From FilesBy default, results are saved to disk in HDF5 format and can be accessed using
theresults_containerattribute ofParallelMap:deff(x,y,z=3):return(x+y)*zwithParallelMap(f,[2,4,6,8],4)aspmap:filenames=pmap.compute()Example loading code:importh5pyimportnumpyasnpout=np.zeros((4,))withh5py.File(pmap.results_container,"r")ash5f:fork,keyinenumerate(h5f.keys()):out[k]=h5f[key]["result_0"][()]See alsoWhere Are My Results?Collect Results in Single HDF5 DatasetIf possible, results can be slotted into a single HDF5 dataset:deff(x,y,z=3):return(x+y)*zwithParallelMap(f,[2,4,6,8],4,result_shape=(None,))aspmap:pmap.compute()Example loading code:importh5pywithh5py.File(pmap.results_container,"r")ash5f:out=h5f["result_0"][()]# returns a NumPy array of shape (4,)More examples can be found inCollect Results in Single DatasetCollect Results in Local MemoryThis is possible but not recommended.deff(x,y,z=3):return(x+y)*zwithParallelMap(f,[2,4,6,8],4,write_worker_results=False)aspmap:result=pmap.compute()# returns a 4-element listAlternatively, create an in-memory NumPy arraywithParallelMap(f,[2,4,6,8],4,write_worker_results=False,result_shape=(None,))aspmap:result=pmap.compute()# returns a NumPy array of shape (4,)DebuggingUse thedebugkeyword to perform all function calls in the local thread of
the active Python interpreterdeff(x,y,z=3):return(x+y)*zwithParallelMap(f,[2,4,6,8],4,z=None)aspmap:results=pmap.compute(debug=True)This way tools likepdbor%debugIPython magics can be used.
More information can be found in theFAQ.Documentation and ContactTo report bugs or ask questions please use ourGitHub issue tracker.
More usage details and background information is available in ouronline documentation.ResourcesACME PresentationatdeRSE23 - Conference for Research Software Engineering in GermanyACME Demopresented at the 4th annual Data Scientist Community MeetingACME TutorialsACME FAQ