Package: simplesimdb
A lightweight simulation data creator and database managerA python module for creation and management of simple simulation data.
Essentially, a very basic database manager that creates its own data.InstallationYou need python3 to install this moduleThe simplest way is to install from the python package indexpypivia the package managerpip:python3-mpipinstallsimplesimdbTo install from github you have to clone the repository and then use the package managerpip.gitclonehttps://github.com/mwiesenberger/simplesimdbcdpath/to/simplesimdb
pipinstall-e.# editable installation of the module# ... if asked, cancel all password prompts ...pytest.# run the unittestsUsageUse this module if you have (i) an executable (the simulation code) that takes a set of input parameters
and generates a single output file and (ii) you want to explore a large region of the
available parameter space ( large here means more than you want to do by hand
but probably less than millions say).
Its tasks aregenerate json input files from given python dictioniaries and assign unique
filenamescall the user provided executable on the input files to create the
corresponding output fileselect existing output files from the data folder based on a given inputdisplay all input parameters for which output data exists in the folderA first exampleHere is an example of how to use the module### example.py ###importjsonimportsimplesimdbimportyamldb=simplesimdb.Manager(directory='ds_data',# where to store the generated datafiletype='yaml',# file-ending of the output filesexecutable='./execute.sh'# the program to execute to generate data)# Let us generate an inputfile for our simulationinputfile={"n":3,"Nx":20,"Ny":20,"Nz":20,"mx":10,"my":10}# run a simulation with the specified input# ./execute.sh ds_data/in.json ds_data/out.yamloutfile=db.create(inputfile)# now open and read the created outputwithopen(outfile)asf:output=yaml.full_load(f)print(json.dumps(output,indent=4))# list all existing inputfiles in the directorycontent=db.table()# the list of inputfiles is searchable:found=list(filter(lambdaentry:(entry["mx"]==10),content))# select a simulation if it existsoutfile=db.select(found[0])# delete all generated data in directory and the directory itselfdb.delete_all()whereexecute.shis an executable that takesdirectory/hashid.jsonas input and creates the filedirectory/hashid.yamlIn this exampleexecute.shparses the json input into command line arguments to a Feltor code and redirect its output into the yaml file.### execute.sh ####!/bin/bash:${FELTOR_PATH:="../feltor"}# Feltor's ds_t requires input parameters from the command line# so we parse the json string using jqcat$1|jq'.n, .Nx, .Ny, .Nz, .mx, .my'|$FELTOR_PATH/inc/geometries/ds_t>$2Human readable namesSometimes, if the number of simulations to run is small a human readable naming
scheme is preferable over the unintuitive sha1 file names. The first
idea would be to name the data directory in a descriptive way. If this
is still not enough, thecreatefunction allows to use a user given name
instead of the sha1 filename. In order for this to work an additional filesimplesimdb.jsonwill be created where the map from sha1 to user name is stored.
For example the above example reads### example.py ###importjsonimportsimplesimdbimportyamldb=simplesimdb.Manager(directory='ds_data',filetype='yaml',executable='./execute.sh')# Let us generate an inputfile for our simulationinputfile={"n":3,"Nx":20,"Ny":20,"Nz":20,"mx":10,"my":10}# Now we choose the name "test" for this simulationoutfile=db.create(inputfile,name="test")If we now inspect theds_datafolder we will find:cdds_data
ls# 'simplesimdb.json' 'test.json' 'test.yaml'The names of the input and output files are now given bytestand we find the
additionalsimplesimddb.json, where the naming schemes are stored, as well. Naming simulations also works on a cluster and when restarting simulations as is described in the following chapters.The following caveats need to be considered:Simulations cannot be renamed except by deleting and resimulatingA name cannot be used more than oncewhen copying data from the repository the "simplesimdb.json" file must also be copied so that simplesimdb can recognize the names in the new folderRunning on a clusterIn many cases simulations are too expensive to run on a local machine. In this case it is mandatory that simulations run on a cluster. The data could then be transfered back to  a local machine where the data exploration with for example a jupyter notebook takes place. The difficulty here is that data is created asynchronously with job submission, i.e. simplesimdb considers the simulation finished when the executable returns even if it just submitted a job to the scheduler and no data was created. Therefore the generation and analysis of data must be separate in this case and the human operator must decide when it is safe to access data and all jobs are finished.With simplesimdb the recommended way of achieving this scenario is:(i) agenerate_data.pyfile on the cluster### generate_data.py ###importjsonimportsimplesimdbassimplesimm=simplesim.Manager(directory='data',executable='./execute.sh',filetype='nc')inputfile=generate_input(...)m.create(inputfile)(ii) theexecute.shscript in this case has to submit the jobs to the job queue (here with slurm):### execute.sh ####!/bin/bashFILE="${2%.*}"# outfilename without extensionJOBNAME=${FILE: -8}# last 8 characters# help simplesimdb recognize that data is being generatedtouch$2# $@ forward all argumentssbatch-o${FILE}.out-J$JOBNAMEsubmit.sh"$@"We touch the output file in order to avoid mistakes where we call the create member twice such that it submits the job twice when the first job is still waiting for resources. This is because simplesimdb checks for the existence of the output file when it decides whether or not a simulation needs to be run in the create member.(iii) thesubmit.shscript is a template job script for all simulations### submit.sh ####!/bin/bash#SBATCH -N 1 -n 1 --ntasks-per-node=1#SBATCH --gres=gpu:1#SBATCH --partition=ppppppppp#SBATCH --account=xxxxx_xxxxx#SBATCH --time=8:00:00:${FELTOR_PATH:="../feltor"}$FELTOR_PATH/src/reco2D/reconnection_hpc$1$2There are a few caveats to this workflow:simplesimdb has no way of knowing when a simulation is pending, running, finished or produced an error. Theexecute.shreturns if the job is successfully submittedin this example all jobs run on exactly the same hardware resources since all use the same job template submit script.Restarting simulationsSometimes simulation outputs cannot be created in a single run (due to file size, time limitations, etc.) but rather a simulation is partitioned (in time) into a sequential number of separate smaller runs. Correspondingly each run sequentially generates and stores only a partition of the entire output file. Each run restarts the simulation with the result of the previous run. The Manager solves this problem via the optional simulation number n in its member functions. For n>0 create will pass the result of the previous simulation as a third input to the executable.See the following example to see how to create and subsequently iterate through restarted simulations### restart_example.py ###importjsonimportsimplesimdbassimplesimm=simplesim.Manager(directory='data',executable='./execute.sh',filetype='txt')# delete content of previous simulation runm.delete_all()# !!! after delete_all() we must reset the directory: !!!m.directory='data'inputfile1={"Hello":"World"}inputfile2={"Hello":"User"}# generate 3 data files for each input:forninrange(0,3):# if n == 0 : ./execute.sh data/in.json data/out_0.txt# if n  > 0 : ./execute.sh data/in.json data/out_n.txt data/out_(n-1).txtm.create(inputfile1,n)m.create(inputfile2,n)# Now search for a specific input file in the contentcontent=m.table()found=list(filter(lambdaentry:(entry["Hello"]=="User"),content))print(found[0])# Count all corresponding simulationsnumber=m.count(found[0])print("Number of simulations found:",number)# and finally loop over and read each of the found simulationsforninrange(0,number):f=open(m.outfile(found[0],n),"r")print("Contents of file",n)print(f.read())Theexecute.shscript in our example does not do much### execute.sh ####!/bin/bashif[$#-eq2]# note the whitespacesthen# do something for first simulationcat$1>$2else# do something else for restartecho"!!!!!RESTART!!!!!">$2cat$3>>$2fiRestarting on a clusterOn the cluster we can also use a combination of simplesimdb a bashexecute.shscript that submits a samplesubmit.shscript. The difficulty is that we do not have the data available immediately and that the submit script needs to find out if previous jobs are running such that it has to wait on.Let us first write a generate scriptgenerate.py(actually this looks the same as before. In fact simplesimdb does not know anything abouthowthe data is generated by the executable)### generate.py ###importjsonimportsimplesimdbassimplesimm=simplesim.Manager(directory='data',executable='./execute.sh',filetype='txt')# delete content of previous simulation runm.delete_all()# after delete_all() we must reset the directorym.directory='data'inputfile1={"Hello":"World"}inputfile2={"Hello":"User"}forninrange(0,3):# if n == 0 : ./execute.sh data/in.json data/out_0.txt# if n  > 0 : ./execute.sh data/in.json data/out_n.txt data/out_(n-1).txtm.create(inputfile1,n)m.create(inputfile2,n)Next we look atexecute.sh### execute.sh ####!/bin/bashFILE="${2%.*}"# outfilename without extensionJOBNAME=${FILE: -8}# last 8 characters# help simplesimdb recognize that data is being generatedtouch$2if[$#-eq2]# note the whitespacesthen# $@ forwards all argumentssbatch-o${FILE}.out-J$JOBNAMEsubmit.sh"$@"else# do something else for restartPREVIOUS="${3%.*}"# previous outfile without extensionPREVNAME=${PREVIOUS: -8}# last 8 characters# check if a job with that name existsJOBID=$(squeue--me--noheader--format="%i"--name"$PREVNAME")# --me only display owned jobs# --noheader suppress the header# --format="%i" only show the jobid# --name only show jobs with specified nameif[-z$JOBID]thensbatch-o${FILE}.out-J$JOBNAMEsubmit.sh"$@"elsesbatch--dependency=afterany:${JOBID}-o${FILE}.out\-J$JOBNAMEsubmit.sh"$@"fifiThis script checks if it is called with 2 or 3 parameters and if it is called with 3 it checks if a job for the previous output already exists. If so it submits a job with a dependency.Thesubmit.shscript in this example does not much:### submit.sh ####!/bin/bash#SBATCH -N 1 -n 1 --ntasks-per-node=1#SBATCH --partition=pppppppp#SBATCH --account=xxxx_xxxx#SBATCH --time=0:10:00 # 24 hours is maximum# Give us a chance to check what jobs are submittedsleep10if[$#-eq2]# note the whitespacesthen# do something for first simulationcat$1>$2else# do something else for restartecho"!!!!!RESTART!!!!!">$2cat$3>>$2fiFinally, when the jobs ran through we can analyse the data### analyse.py ###importjsonimportsimplesimdbassimplesimm=simplesim.Manager(directory='data',executable='./execute.sh',filetype='txt')content=m.table()found=list(filter(lambdaentry:(entry["Hello"]=="User"),content))print(found[0])number=m.count(found[0])print("Number of simulations found:",number)forninrange(0,number):f=open(m.outfile(found[0],n),"r")print("Contents of file",n)print(f.read())The result is the same as in the previous sectionDiagnosticsSometimes, for very large simulations, we want to run post-processing diagnostics that is also written in another language. This post-processing takes the form of./diag output.nc diag_output.ncIn order to integrate well with ourManagerclass we suggest to keep simulation data and diagnostics data in separate folders (for example "data" and "diag"). Then, create a second Manager on the "diag" folder and make theexecute.shscript (i) ignore the input and restart files and (ii) in the output file replace the "diag" folder with the original "data" folder and use that one as input:### diag.sh ####!/bin/bash:${FELTOR_PATH:="../feltor"}# we change the diag folder to data (assuming these are the folder names in use)input=$(echo$2|sed-e's/diag/data/')# ignore $1 (the input.json) and $3 (the restart file)$FELTOR_PATH/src/feltor/feltordiag$input$2In python we can then create and search diagnostic files by input parameters and simulation number as usual:### diagnose.py ###importjsonimportsimplesimdbassimplesimdata=simplesim.Manager(directory='data',filetype='nc')diag=simplesim.Manager(directory="diag",filetype="nc",executable="./diag.sh")content=data.table()found=list(filter(lambdaentry:(entry["Hello"]=="World"),content))number=data.count(found[0])print("Number of simulations found:",number)forninrange(0,number):# Simulation dataoutfile=data.select(found[0],n)# corresponding diagnosticsdiagfile=diag.create(found[0],n)f=open(diagfile,"r")# ... do stuffRepeaterSometimes we do not want to store simulation data on disc and it is enough to simply store temporary files. This can for example happen when simulations run very quickly and data should be created "on demand". For such a case we have theRepeaterclass, which simply overwrites the input and output for each new simulation:### geometry_diag.sh ####!/bin/bash:${FELTOR_PATH:="../feltor"}make-C$FELTOR_PATH/inc/geometriesgeometry_diagdevice=omp

rm-f$2# in case the file is opened elsewhere ...$FELTOR_PATH/inc/geometries/geometry_diag$1$2### calibrate.py ###importjsonimportsimplesimdbassimplesimrep=simplesim.Repeater("./execute.sh","temp.json","temp.nc")inputdata={"Hello":"World"}rep.run(inputdata,error="display",stdout="ignore")# ... do something with "temp.nc"inputdata={"Hello":"User"}# Overwrite both "temp.json" and "temp.nc"rep.run(inputdata,error="display",stdout="ignore")# ... do something else with "temp.nc"rep.clean()The trick to get a diagnostics to work is similar as above### short-diagnose.py ###importjsonimportsimplesimdbassimplesim# by naming it data and diag we get the diag.sh from above to workrep=simplesim.Repeater("./execute.sh","data.json","data.nc")diag=simplesim.Repeater("./diag.sh","diag.json","diag.nc")inputdata={"Hello":"World"}rep.run(inputdata,error="display",stdout="ignore")# ... do something with "data.nc"diag.run(inputdata)# ... do something with "diag.nc"rep.clean()diag.clean()CaveatsCannot manage simulations with more than one input file
(but you can write a shell script that simple adds the second input file)Cannot manage simulations with more than one output file
(again, with a script you can add a second output file)Cannot manage existing simulation files that do not have names assigned by the module
(You can register names manually with theregisterfunction)Do not capture the stdout and stderr streams of the executable.
(It is recommended that you redirect these streams into a file output yourself if you need it)simplesimdb considers a simulation successful if the output file exists. It cannot realize whether the content of the file is sane or corrupt, or whether there is any content at all for that matterWhy not just use an existing database management software like SQLRelational databases are notoriously unsuited for scientific simulations so
SQL is mostly not an option in the scientific community (in python we would
use pandas anyway)Search methods that DBMS offer are offered by python for dictionaries as
well (especially if they are converted to tables in pandas)There is an additional overhead of setting up and running a databaser serverWhen doing a parameter study you do not deal with millions of entries but a
only a few hundred at best so a DBMS might just be overkillThere are NoSQL database managers like for example a document (json) manager
like MongoDB that can  manage large files (videos, images, PDFs, etc.) along
with supporting information (metadata) that fits naturally into JSON
documents, butNetcdf (or other output) files do not really map nicely into the mongodb
data model since files larger than 16 MB require another FileGS server to be
runIf netcdf files are stored by the DB manager how do we get access to it
through other external programs like for example paraview?If json files are hidden by the database manager how do we use them as input
to run an executable to generate the output file?ContributionsContributions are welcome.AuthorsMatthias Wiesenberger