Package: chastesweep
Chaste Parameter SweeperA tool to assist with parameter sweeping forChasteon the HPC (SGE & Slurm). The package provides parameter expansion, generation of task array batch script, individual job runner and a c++ main() function generator.The parameter sweeping code was brought in from thepscanpackage and converted to python 2.x to ensure it works with the rest of Chaste's build dependencies.While originally created for Chaste, it can also be applied to any other application that correctly accepts the parameters in the correct format (i.e. your program should accept parameters in the format of--varname value).InstallationBy pipInstall by using:pipinstallchastesweepFrom repository# Clone the respository then install itgitclonehttps://github.com/twinkarma/chastesweep.gitcdchastesweep
pythonsetup.pyinstallGeneral UsageAfter installing thechastesweeppackage, create a python script for defining your sweep parameters e.g. we create asweep.pyfile:importnumpyasnpfromchastesweepimportParamSweeper# We'll use 3 parameters in this example, param0, param1 and param2# we create a dictionary and declare what each parameter values arep={}p['param0']=np.linspace(0,10,5)p['param1']=np.linspace(0.1,-0.5,5)p['param2']=[10,20,30]# We must also declare the execuatable that will run with the paramtersexec_cmd="my_executable.sh"# And the output directoryoutput_dir="sweep_results"# We then create the parameter sweeper and have it generate the batch file in the output directorysweeper=ParamSweeper()sweeper.generate_batch_output(output_dir=output_dir,exec_cmd=exec_cmd,parameters=p,scheduler=ParamSweeper.SGE,batch_params=["-M myemail@mydomain.com","-m bes"])We then runsweep.py:pythonsweep.pyIn the output directorysweep_results, you will see theparams.json,runsimulation.pyandbatch.sge.shfile.params.jsonContains an expanded list of parameters that will be explored.batch.sge.shBatch script containing a task array for running through all of the parameters to be explored.runsimulation.pySimulation runner script for running individual instances of the simulation.You can submit this file to the SGE scheduler to start your parameter sweeping task:qsubsweep_results/batch.sge.shIf we're running a SLURM scheduler, simply change thescheduler=ParamSweeper.SGEtoscheduler=ParamSweeper.SLURM. You'll also want to change yourbatch_paramsparameters as they'll likely be different from SGE. Your batch submit command will also change to:sbatchsweep_results/batch.slurm.shExpanding parametersThe following is a demonstration of how parameters can be expanded. All examples also apply togenerate_batch_output.To only get the parameter expansion, call theexpand_parametersfunction:importnumpyasnpfromchastesweepimportParamSweeper# We'll use 3 parameters in this example, param0, param1 and param2# we create a dictionary and declare what each parameter values arep={}p['param0']=np.linspace(0,10,5)p['param1']=np.linspace(0.1,-0.5,5)p['param2']=[10,20,30]# We can then call the expsweeper=ParamSweeper()results=sweeper.expand_parameters(parameters=p)Joining ParametersBy default, parameters are combinatorially expanded. To prevent this expansion between some parameters, the parameters can be joined together by providing ajoinlist. For example if you'd like to joinparam0andparam1in the above example:# We create a join listjoin_lists=[['param0','param1']]# Add the join list to the call parametersresults=sweeper.expand_parameters(parameters=p,joint_lists=join_lists)You'll see that ourresultslist has15items instead of75if we didn't include the join list.Globally running each parameter set more than onceYou can set thedefault_repeatsto specify globally how many iterations of each parameter set will run, e.g. the value of2:results=sweeper.expand_parameters(parameters=p,default_repeats=2)Result in150simulation runs.Variable number of iteration for each parameter setInstead of setting the number of iterations globally, the number of iteration per parameter set can vary by declaring count functions:# The default count function, runs 2 iterations of every parameterdefault_count=lambdap:2# A big_count function that forces the iteration to 1 if param 2 is larger than 13big_count=lambdap:1ifp['param2']>=14elseNone# The order of counters makes a difference, the next counter in the sequence can overwrite previous countscount_funcs=[default_count,big_count]results=sweeper.expand_parameters(parameters=p,count_funcs=count_funcs)Running the sweep locallySweep can also be run locally on your machine with theperform_serial_sweepfunction:importnumpyasnpfromchastesweepimportParamSweeperp={}p['a']=np.linspace(0,10,5)p['b']=np.linspace(0.1,-0.5,5)p['c']=[10,20,30]sweeper=ParamSweeper()exec_cmd="my_executable.sh"output_dir="sweep_results"sweeper.perform_serial_sweep(output_dir=output_dir,exec_cmd=exec_cmd,parameters=p)Parameter Sweeping Tutorial (Sheffield HPC)In this tutorial we will go through how to setup chaste for parameter sweeping on the HPC cluster. Run this tutorial directly on the cluster to be able to follow all examples including job submission.1. Getting a compute nodeWhen logging in to ShARC or Bessemer, you'll start off on a login node. You'll want to get an interactive session on a worker node to start development:# On ShARCqrshx# Bessemersrun--ptybash-iYou'll see the console change fromlogintonodee.g.[username@sharc-login#] $to[username@sharc-node###] $in the case of ShARC.2. Install chastesweep python packageOn ShARCIt is recommended to create a virtual environment on ShARC:# Load the python modulemoduleloadapps/python/anaconda3-4.2.0# Create a new virtual python environment called my_chaste_env with python 2.7condacreate-nmy_chaste_envpython=2.7# Activate the environmentsourceactivatemy_chaste_env# Install the chastesweep packagepipinstallchastesweepOn BessemerEquivalent code on Bessemer# Load the python modulemoduleloadAnaconda3/5.3.0# Create a new virtual python environment called my_chaste_env with python 2.7condacreate-nmy_chaste_env2python=2.7chastesweep# Activate the environmentsourceactivatemy_chaste_env2# Install the chastesweep packagepipinstallchastesweep3. Development Environment, pulling a a Singularity imageShARC and Bessemer supports theSingularitycontainerisation technology. The use of containers allow us to standardise the development environment and build dependencies on your local machine and your cluster.Note: On your local machine, if you're not developing on Linux, it is also possible to use docker instead. You will want to install eitherSingularityordockeron your local machine.To get the latest Singularity image of Chaste, run the following:singularitypulldocker://chaste/chaste-docker:latestYou should see a filechaste-docker_latest.sif.Run the following to get into the image's bash shell:singularityexecchaste-docker_latest.sif/bin/bashYou're now actually inside the image, try running the commandcat /etc/lsb-releaseto see the current Ubuntu version e.g.:cat/etc/lsb-releaseDISTRIB_ID=UbuntuDISTRIB_RELEASE=19.04DISTRIB_CODENAME=discoDISTRIB_DESCRIPTION="Ubuntu 19.04"You can runexitto exit from the image.Singularity'sexeccommand can be used to run any program from inside the image without needing to go into the interactive session. This is useful when we're running our tasks in a batch script. For example, to get the Ubuntu version as above:singularityexecchaste-docker_latest.sifcat/etc/lsb-releaseYou will be using the image you've just pulled to do the building and running of the code using theexeccommand as shown above.Note: As you will only be using the container for its build dependency, you will not need to modify the image during development and so should not need admin permission.4. Preparing your codeYou now need your own version of Chaste code. The Chaste repository can be found athttps://github.com/Chaste/Chasteand this can be forked or cloned directly depending on your requirements. We will just clone directly in this example:# Clone the Chaste source code into our home foldergitclonehttps://github.com/Chaste/Chaste.git~/Chaste# Go inside the foldercd~/ChasteThe above command clones Chaste code into theChastefolder to your home directory. You will now need create a custom Chaste app or project with a main() function to be able to accept parameters.The the instructions on how to create  Chaste executable apps and user project can be followed below:Make executable apps:https://chaste.cs.ox.ac.uk/trac/wiki/ChasteGuides/BuildingExecutableAppsUser projects:https://chaste.cs.ox.ac.uk/trac/wiki/ChasteGuides/UserProjectsFor this example we'll go with the executable app route due to the simpler setup. We'll ask the tool to generate the.cppfile for us.We'll call our appParamSweepand will use three parameters for this example, namedparam0,param1andparam2. To automatically generate a template using the parameter sweeper tool, call:chastesweep_genmainparam0,param1,param2apps/src/ParamSweep.cppThe generated source code is printed on screen as well as created for you.We can then try to build the project usingcmake:# Go into the build directorycdbuild# Generate makefiles using cmake from inside the Chaste imagesingularityexecchaste-docker_latest.sifcmake..# Build our ParamSweep app from inside the Chaste imagesingularityexecchaste-docker_latest.sifmakeParamSweepAfter the build has finished, we can now run our app:singularityexecchaste-docker_latest.sifapps/ParamSweepYour code should execute as is but you'll get errors stating that parameters are missing.5. Passing parameters to the main functionThe parameter sweeper follows the boost::program_options format, where each parameter is passed in the format of--param_name param_valuee.g. for ourParamSweepapp:singularityexecchaste-docker_latest.sifapps/ParamSweep--output_dir"/outtput/path/1"--param00.2--param13--param2-0.53Note: Theoutput_dirparameter is always passed to your app and all outputs of the app should be enforced to only write to this directory.6. Declaring parameters and generating the batch scriptNow that we have an executable ready for accepting parameters, we'll move on to declaring the values of parameters to be explored and generating batch script for submitting a batch task. For this purpose, python API has been created, the classParamSweeperhas been created to facilitate this parameter sweeping process.We create a python script namedsweep.pyfor defining our parameter and creating a batch job:importnumpyasnpfromchastesweepimportParamSweeperp={}p['param0']=np.linspace(0,10,5)p['param1']=np.linspace(0.1,-0.5,5)p['param2']=[10,20,30]exec_cmd="~/Chaste/build/apps/ParamSweep"output_dir="~/Chaste/build/sweep_results"# Generate batch for SGEsweeper=ParamSweeper()sweeper.generate_batch_output(output_dir=output_dir,exec_cmd=exec_cmd,parameters=p,scheduler=ParamSweeper.SGE,batch_params=["-M myemail@mydomain.com","-m bes"])Notice that we can additional parameters to the batch script by adding tobatch_params. In our example we've set the scheduler to alert us of the job status-m besand provided our e-mail-M myemail@mydomain.com.We then run thesweep.pyscript:pythonsweep.pyIn the output directory~/Chaste/build/sweep_results, you will see theparams.json,runsimulation.pyandbatch.sge.shfile.params.jsonContains an expanded list of parameters that will be explored.batch.sge.shBatch script containing a task array for running through all of the parameters to be explored.runsimulation.pySimulation runner script for running individual instances of the simulation.You can submit this file to the SGE scheduler to start your parameter sweeping task:qsub~/Chaste/build/sweep_results/batch.sge.shIf we're running on Bessemer or other cluster that uses SLURM scheduler, simply change thescheduler=ParamSweeper.SGEtoscheduler=ParamSweeper.SLURM. You'll also want to change yourbatch_paramsparameters as they'll likely be different from SGE. Your batch submit command will also change to:sbatch~/Chaste/build/sweep_results/batch.slurm.sh