Package: colmena
ColmenaColmena simplifies building autonomous applications that steer large campaigns of simulations on supercomputers.Such "high-throughput" campaigns were, historically,
guided by humans identifying the which tasks to run next — a time-consuming process with a high latency between "new data" and "decisions."Colmena was created to build applications which augment or replace human steering with Artificial Intelligence (AI).InstallationColmena is availablevia PyPI:pip install colmenaConsult ourInstallation Guidefor further details.Using ColmenaColmena applications describe a computational campaign in two components: a "Thinker" that picks computations
and a "Doer" which executes them.Thinkers encode the logic for how to run new calculations as "agents."
Complex strategies are simple to express when you decompose them into simple steps.
For example, a distributed optimizer:fromrandomimportrandomfromcolmena.thinkerimportBaseThinker,result_processor,task_submitter,ResourceCounterfromcolmena.queueimportPipeQueuesfromcolmena.modelsimportResult# Build queues to connect Thinker and Doerqueues=PipeQueues()classThinker(BaseThinker):def__init__(self,queues,num_workers:int,num_guesses=100):super().__init__(queues,ResourceCounter(num_workers))self.best_result=Noneself.answer=-10# A (bad) starting guessself.num_guesses=num_guesses@task_submitter()defsubmit_task(self):"""Submit a new guess close to the current best whenever a node is free"""self.queues.send_inputs(self.answer-1+2*random(),method='simulate')@result_processor()defstore_result(self,result:Result):"""Update best guess whenever a simulation finishes"""assertresult.success,result.failure_info# Update the best resultifself.best_resultisNoneorresult.value>self.best_result:self.answer=result.args[0]self.best_result=result.valueself.rec.release()# Mark that a node is now free# Determine if we are doneself.num_guesses-=1ifself.num_guesses<=0:self.done.set()thinker=Thinker(queues,8)Doers describe the types of computations and available compute resources.
Colmena provides Task Servers backed by several workflow engines, such as those from theExaWorks project.
Building one using Parsl requires only that your computations are expressed as Python functions:fromparsl.configs.htex_localimportconfig# Configuration to run locallyfromcolmena.task_serverimportParslTaskServer# Define your functiondefsimulate(x:float)->float:return-x**2+4# Make the Doerdoer=ParslTaskServer([simulate],queues,config)Once these are defined, launching the application involves starting both# Launch the Thinker and doerdoer.start()thinker.start()# Wait until it finishesthinker.join()queues.send_kill_signal()# Stop the doer# Done!print(f'Answer: f({thinker.answer:.2f}) ={thinker.best_result:.2f}')TutorialsVisit theQuickstartto learn to build a full application.More ExamplesSee thedemo_appsto see a variety of ways to use Colmena.Learning MoreOurRead-the-Docsprovides the most up-to-date information about Colmena.You can also learn more about Colmena in the papers we published about it:Ward et al. "Colmena: Scalable Machine-Learning-Based Steering of Ensemble Simulations for High Performance Computing".2021 IEEE/ACM Workshop on Machine Learning in High Performance Computing Environments (MLHPC)[doi] [ArXiv]
[slides]
[YouTube]AcknowledgementsThis project was supported in part by the Exascale Computing Project (17-SC-20-SC) of the U.S. Department of Energy (DOE) and by DOE’s Advanced Scientific Research Office (ASCR) under contract DE-AC02-06CH11357.