Package: smartsim
HomeInstallDocumentationCray LabsContactJoin us on Slack!SmartSimSmartSim is made up of two partsSmartSim Infrastructure Library (This repository)SmartRedisThe two library components are designed to work together, but can also be used
independently.SmartSimis a workflow library that makes it easier to use common Machine Learning (ML)
libraries, like PyTorch and TensorFlow, in High Performance Computing (HPC) simulations
and applications. SmartSim launches ML infrastructure on HPC systems alongside user
workloads.SmartRedisprovides an API to connect HPC workloads, particularly (MPI + X) simulations,
to the ML infrastructure, namely the TheOrchestrator database,
launched by SmartSim.Applications integrated with the SmartRedis clients, written in Fortran, C, C++ and Python,
can send data to and remotely request SmartSim infrastructure to execute ML models and scripts
on GPU or CPU. The distributed Client-Server paradigm allows for data to be seamlessly
exchanged between applications at runtime without the utilization of MPI.Table of ContentsSmartSimQuick StartSmartSim Infrastructure LibraryExperimentsHello WorldHello World MPIExperiments on HPC SystemsInteractive Launch ExampleBatch Launch ExamplesInfrastructure Library ApplicationsRedis + RedisAILocal LaunchInteractive LaunchBatch LaunchSmartRedisTensorsDatasetsSmartSim + SmartRedis TutorialsRun the TutorialsOnline AnalysisLattice Boltzmann SimulationOnline ProcessingSingular Value DecompositionOnline InferencePyTorch CNN ExamplePublicationsCitebibtexQuick StartThe documentation has a number of tutorials that make it easy to get used to SmartSim locally
before using it on your system. Each tutorial is a Jupyter notebook that can be run through theSmartSim Tutorials docker imagewhich will run a jupyter lab with the tutorials, SmartSim, and SmartRedis installed.dockerpullghcr.io/craylabs/smartsim-tutorials:latest
dockerrun-p8888:8888ghcr.io/craylabs/smartsim-tutorials:latest# click on link to open jupyter labSmartSim Infrastructure LibraryThe Infrastructure Library (IL), thesmartsimpython package,
facilitates the launch of Machine Learning and simulation
workflows. The Python interface of the IL creates, configures, launches and monitors
applications.ExperimentsTheExperimentobject
is the main interface of SmartSim. Through theExperimentusers can create references to user applications calledModels.Hello WorldBelow is a simple example of a workflow that uses the IL to launch hello world
program using the local launcher which is designed for laptops and single nodes.fromsmartsimimportExperimentexp=Experiment("simple",launcher="local")settings=exp.create_run_settings("echo",exe_args="Hello World")model=exp.create_model("hello_world",settings)exp.start(model,block=True)print(exp.get_status(model))Hello World MPITheExperiment.create_run_settingsmethod returns aRunSettingsobject which
defines how a model is launched. There are many types ofRunSettingssupported by
SmartSim.RunSettingsMpirunSettingsSrunSettingsAprunSettingsJsrunSettingsThe following example launches a hello world MPI program using the local launcher
for single compute node, workstations and laptops.fromsmartsimimportExperimentexp=Experiment("hello_world",launcher="local")mpi_settings=exp.create_run_settings(exe="echo",exe_args="Hello World!",run_command="mpirun")mpi_settings.set_tasks(4)mpi_model=exp.create_model("hello_world",mpi_settings)exp.start(mpi_model,block=True)print(exp.get_status(model))If an argument ofrun_command="auto"(the default) is passed toExperiment.create_run_settings, SmartSim will attempt to find a run command on the
system with which it has a correspondingRunSettingsclass. If one can be found,Experiment.create_run_settingswill instance and return an object of that type.Experiments on HPC SystemsSmartSim integrates with common HPC schedulers providing batch and interactive
launch capabilities for all applications.SlurmLSFPBSProLocal (for laptops/single node, no batch)Interactive Launch ExampleThe following launches the samehello_worldmodel in an interactive allocation.# get interactive allocation (Slurm)salloc-N3--ntasks-per-node=20--ntasks60--exclusive-t00:10:00# get interactive allocation (PBS)qsub-lselect=3:ncpus=20-lwalltime=00:10:00-lplace=scatter-I-q<queue># get interactive allocation (LSF)bsub-Is-W00:10-nnodes3-P<project>$SHELLThis same script will run on a SLURM, PBS, or LSF system as thelauncheris set toautoin theExperimentinitialization. The run command likempirun,aprunorsrunwill be automatically detected from what is available on the
system.# hello_world.pyfromsmartsimimportExperimentexp=Experiment("hello_world_exp",launcher="auto")run=exp.create_run_settings(exe="echo",exe_args="Hello World!")run.set_tasks(60)run.set_tasks_per_node(20)model=exp.create_model("hello_world",run)exp.start(model,block=True,summary=True)print(exp.get_status(model))# in interactive terminalpythonhello_world.pyThis script could also be launched in a batch file instead of an
interactive terminal. For example, for Slurm:#!/bin/bash#SBATCH --exclusive#SBATCH --nodes=3#SBATCH --ntasks-per-node=20#SBATCH --time=00:10:00python/path/to/hello_world.py# on Slurm systemsbatchrun_hello_world.shBatch Launch ExamplesSmartSim can also launch workloads in a batch directly from Python, without the need
for a batch script. Users can launch groups ofModelinstances in aEnsemble.The following launches 4 replicas of the the samehello_worldmodel.# hello_ensemble.pyfromsmartsimimportExperimentexp=Experiment("hello_world_batch",launcher="auto")# define resources for all ensemble membersbatch=exp.create_batch_settings(nodes=4,time="00:10:00",account="12345-Cray")batch.set_queue("premium")# define how each member should runrun=exp.create_run_settings(exe="echo",exe_args="Hello World!")run.set_tasks(60)run.set_tasks_per_node(20)ensemble=exp.create_ensemble("hello_world",batch_settings=batch,run_settings=run,replicas=4)exp.start(ensemble,block=True,summary=True)print(exp.get_status(ensemble))pythonhello_ensemble.pySimilar to the interactive example, this same script will run on a SLURM, PBS,
or LSF system as thelauncheris set toautoin theExperimentinitialization. Local launching does not support batch workloads.Infrastructure Library ApplicationsOrchestrator - In-memory data store and Machine Learning Inference (Redis + RedisAI)Redis + RedisAITheOrchestratoris an in-memory database that utilizes Redis and RedisAI to provide
a distributed database and access to ML runtimes from Fortran, C, C++ and Python.SmartSim provides classes that make it simple to launch the database in many
configurations and optionally form a distributed database cluster. The examples
below will show how to launch the database. Later in this document we will show
how to use the database to perform ML inference and processing.Local LaunchThe following script launches a single database using the local launcher.Experiment.create_databasewill initialize anOrchestratorinstance corresponding to the specified launcher.# run_db_local.pyfromsmartsimimportExperimentexp=Experiment("local-db",launcher="local")db=exp.create_database(port=6780,# database portinterface="lo")# network interface to use# by default, SmartSim never blocks execution after the database is launched.exp.start(db)# launch models, analysis, training, inference sessions, etc# that communicate with the database using the SmartRedis clients# stop the databaseexp.stop(db)Interactive LaunchTheOrchestrator, likeEnsembleinstances, can be launched locally, in interactive
allocations, or in a batch.The following example launches a distributed (3 node) database cluster
in an interactive allocation.# get interactive allocation (Slurm)salloc-N3--ntasks-per-node=1--exclusive-t00:10:00# get interactive allocation (PBS)qsub-lselect=3:ncpus=1-lwalltime=00:10:00-lplace=scatter-I-qqueue# get interactive allocation (LSF)bsub-Is-W00:10-nnodes3-Pproject$SHELL# run_db.pyfromsmartsimimportExperiment# auto specified to work across launcher typesexp=Experiment("db-on-slurm",launcher="auto")db_cluster=exp.create_database(db_nodes=3,db_port=6780,batch=False,interface="ipogif0")exp.start(db_cluster)print(f"Orchestrator launched on nodes:{db_cluster.hosts}")# launch models, analysis, training, inference sessions, etc# that communicate with the database using the SmartRedis clientsexp.stop(db_cluster)# in interactive terminalpythonrun_db.pyBatch LaunchTheOrchestratorcan also be launched in a batch without the need for an interactive allocation.
SmartSim will create the batch file, submit it to the batch system, and then wait for the database
to be launched. Users can hit CTRL-C to cancel the launch if needed.# run_db_batch.pyfromsmartsimimportExperimentexp=Experiment("batch-db-on-pbs",launcher="auto")db_cluster=exp.create_database(db_nodes=3,db_port=6780,batch=True,time="00:10:00",interface="ib0",account="12345-Cray",queue="cl40")exp.start(db_cluster)print(f"Orchestrator launched on nodes:{db_cluster.hosts}")# launch models, analysis, training, inference sessions, etc# that communicate with the database using the SmartRedis clientsexp.stop(db_cluster)pythonrun_db_batch.pySmartRedisThe SmartSim IL Clients (SmartRedis)
are implementations of Redis clients that implement the RedisAI
API with additions specific to scientific workflows.SmartRedis clients are available in Fortran, C, C++, and Python.
Users can seamlessly pull and push data from the Orchestrator from different languages.TensorsTensors are the fundamental data structure for the SmartRedis clients. The Clients
use the native array format of the language. For example, in Python, a tensor is
a NumPy array while the C/C++ clients accept nested and contiguous arrays.When stored in the database, all tensors are stored in the same format. Hence,
any language can receive a tensor from the database no matter what supported language
the array was sent from. This enables applications in different languages to communicate
numerical data with each other at runtime.For more information on the tensor data structure, seethe documentationDatasetsDatasets are collections of Tensors and associated metadata. TheDatasetclass
is a user space object that can be created, added to, sent to, and retrieved from
the Orchestrator.For an example of how to use theDatasetclass, see theOnline Analysis exampleFor more information on the API, see theAPI documentationSmartSim + SmartRedis TutorialsSmartSim and SmartRedis were designed to work together. When launched through
SmartSim, applications using the SmartRedis clients are directly connected to
any Orchestrator launched in the sameExperiment.In this way, a SmartSimExperimentbecomes a driver for coupled ML and Simulation
workflows. The following are simple examples of how to use SmartSim and SmartRedis
together.Run the TutorialsEach tutorial is a Jupyter notebook that can be run through theSmartSim Tutorials docker imagewhich will run a jupyter lab with the tutorials, SmartSim, and SmartRedis installed.dockerpullghcr.io/craylabs/smartsim-tutorials:latest
dockerrun-p8888:8888ghcr.io/craylabs/smartsim-tutorials:latestEach of the following examples can be found in theSmartSim documentation.Online AnalysisUsing SmartSim, HPC applications can be monitored in real time by streaming data
from the application to the database. SmartRedis clients can retrieve the
data, process, analyze it, and finally store any updated data back to the database for
use by other clients.The following is an example of how a user could monitor and analyze a simulation.
The example here uses the Python client; however, SmartRedis clients are also available
for C, C++, and Fortran. All SmartRedis clients implement the same API.The example will producethis visualizationwhile the simulation is running.Lattice Boltzmann SimulationUsing aLattice Boltzmann Simulation,
this example demonstrates how to use the SmartRedisDatasetAPI to stream
data over the Orchestrator deployed by SmartSim.The simulation will be composed of two parts:fv_sim.pywhich will generate data from
the Simulation and store it in the Orchestrator, anddriver.pywhich will launch the Orchestrator, startfv_sim.pyand check for data posted to the
Orchestrator to plot updates in real-time.The following code highlights the sections offv_sim.pythat are responsible for
transmitting the data needed to plot timesteps of the simulation to the Orchestrator.# fv_sim.pyfromsmartredisimportClientimportnumpyasnp# initialization code omitted# save cylinder location to databasecylinder=(X-x_res/4)**2+(Y-y_res/2)**2<(y_res/4)**2# bool arrayclient.put_tensor("cylinder",cylinder.astype(np.int8))fortime_stepinrange(steps):# simulation loopfori,cx,cyinzip(idxs,cxs,cys):F[:,:,i]=np.roll(F[:,:,i],cx,axis=1)F[:,:,i]=np.roll(F[:,:,i],cy,axis=0)bndryF=F[cylinder,:]bndryF=bndryF[:,[0,5,6,7,8,1,2,3,4]]rho=np.sum(F,2)ux=np.sum(F*cxs,2)/rhouy=np.sum(F*cys,2)/rhoFeq=np.zeros(F.shape)fori,cx,cy,winzip(idxs,cxs,cys,weights):Feq[:,:,i]=rho*w*(1+3*(cx*ux+cy*uy)+9*(cx*ux+cy*uy)**2/2-3*(ux**2+uy**2)/2)F+=-(1.0/tau)*(F-Feq)F[cylinder,:]=bndryF# Create a SmartRedis dataset with vorticity datadataset=Dataset(f"data_{str(time_step)}")dataset.add_tensor("ux",ux)dataset.add_tensor("uy",uy)# Put Dataset in db at key "data_{time_step}"client.put_dataset(dataset)The driver script,driver.py, launches the Orchestrator database and runs
the simulation in a non-blocking fashion. The driver script then uses the SmartRedis
client to pull the DataSet and plot the vorticity while the simulation is running.# driver.pytime_steps,seed=3000,42exp=Experiment("finite_volume_simulation",launcher="local")db=exp.create_database(port=6780,# database portinterface="lo")# network interface db should listen on# create the lb simulation Model referencesettings=exp.create_run_settings("python",exe_args=["fv_sim.py",f"--seed={seed}",f"--steps={time_steps}"])model=exp.create_model("fv_simulation",settings)model.attach_generator_files(to_copy="fv_sim.py")exp.generate(db,model,overwrite=True)exp.start(db)client=Client(address=db.get_address()[0],cluster=False)# start simulation (non-blocking)exp.start(model,block=False,summary=True)# poll until simulation starts and then retrieve dataclient.poll_key("cylinder",200,100)cylinder=client.get_tensor("cylinder").astype(bool)foriinrange(0,time_steps):client.poll_key(f"data_{str(i)}",10,1000)dataset=client.get_dataset(f"data_{str(i)}")ux,uy=dataset.get_tensor("ux"),dataset.get_tensor("uy")# analysis/plotting code omittedexp.stop(db)For more examples of how to use SmartSim and SmartRedis together to perform
online analysis, please see theonline analsysis tutorial sectionof the
SmartSim documentation.Online ProcessingEach of the SmartRedis clients can be used to remotely executeTorchScriptcode on data
stored within the database. The scripts/functions are executed in the Torch
runtime linked into the database.Any of the functions available in theTorchScript builtinscan be saved as "script" or "functions" in the database and used directly by
any of the SmartRedis Clients.Singular Value DecompositionFor example, the following code sends the built-inSingular Value Decompositionto the database and execute it on a dummy tensor.importnumpyasnpfromsmartredisimportClient# don't even need to import torchdefcalc_svd(input_tensor):returninput_tensor.svd()# connect a client to the databaseclient=Client(cluster=False)# get dummy datatensor=np.random.randint(0,100,size=(5,3,2)).astype(np.float32)client.put_tensor("input",tensor)client.set_function("svd",calc_svd)client.run_script("svd","calc_svd","input",["U","S","V"])# results are not retrieved immediately in case they need# to be fed to another function/modelU=client.get_tensor("U")S=client.get_tensor("S")V=client.get_tensor("V")print(f"U:{U}, S:{S}, V:{V}")The processing capabilities make it simple to form computational pipelines of
functions, scripts, and models.See the fullTorchScript Language Referencedocumentation for more information on available methods, functions, and how
to create your own.Online InferenceSmartSim supports the following frameworks for querying Machine Learning models
from C, C++, Fortran and Python with the SmartRedis Clients:RedisAI VersionLibrariesSupported Version1.2.7PyTorch2.0.1TensorFlow\Keras2.13.1ONNX1.14.1Anumber of other librariesare
supported through ONNX, likeSciKit-LearnandXGBoost.Note:It's important to remember that SmartSim utilizes a client-server model. To run
experiments that utilize the above frameworks, you must first start the Orchestrator
database with SmartSim.PyTorch CNN ExampleThe example below shows how to spin up a database with SmartSim and
invoke a PyTorch CNN model using the SmartRedis clients.# simple_torch_inference.pyimportioimporttorchimporttorch.nnasnnfromsmartredisimportClientfromsmartsimimportExperimentexp=Experiment("simple-online-inference",launcher="local")db=exp.create_database(port=6780,interface="lo")classNet(nn.Module):def__init__(self):super().__init__()self.conv=nn.Conv2d(1,1,3)defforward(self,x):returnself.conv(x)torch_model=Net()example_forward_input=torch.rand(1,1,3,3)module=torch.jit.trace(torch_model,example_forward_input)model_buffer=io.BytesIO()torch.jit.save(module,model_buffer)exp.start(db,summary=True)address=db.get_address()[0]client=Client(address=address,cluster=False)client.put_tensor("input",example_forward_input.numpy())client.set_model("cnn",model_buffer.getvalue(),"TORCH",device="CPU")client.run_model("cnn",inputs=["input"],outputs=["output"])output=client.get_tensor("output")print(f"Prediction:{output}")exp.stop(db)The above python code can be run like any normal python script:pythonsimple_torch_inference.pyFor more examples of how to use SmartSim and SmartRedis together to perform
online inference, please see theonline inference tutorials sectionof the
SmartSim documentation.PublicationsThe following are public presentations or publications using SmartSimCollaboration with NCAR - CGD SeminarSmartSim: Using Machine Learning in HPC SimulationsSmartSim: Online Analytics and Machine Learning for HPC SimulationsPyTorch Ecosystem Day PosterCitePlease use the following citation when referencing SmartSim, SmartRedis, or any SmartSim related work:Partee et al., “Using Machine Learning at Scale in HPC Simulations with SmartSim:
An Application to Ocean Climate Modeling”, Journal of Computational Science, Volume 62, 2022, 101707, ISSN 1877-7503Available:https://doi.org/10.1016/j.jocs.2022.101707.bibtex@article{PARTEE2022101707,
    title ={Using Machine Learning at scale in numerical simulations with SmartSim: An application to ocean climate modeling},
    journal ={Journal of Computational Science},
    volume ={62},
    pages ={101707},
    year ={2022},
    issn ={1877-7503},
    doi ={https://doi.org/10.1016/j.jocs.2022.101707},
    url ={https://www.sciencedirect.com/science/article/pii/S1877750322001065},
    author ={Sam Partee and Matthew Ellis and Alessandro Rigazzi and Andrew E. Shao and Scott Bachman and Gustavo Marques and Benjamin Robbins},
    keywords ={Deep learning, Numerical simulation, Climate modeling, High performance computing, SmartSim},}