Package: blackdynamite
Quick startDocumentationThe complete documentation of the project is onreadthedocsInstallationThe easiest is through pip:pipinstallblackdynamiteFor a user scope installation (recommended):pipinstall--userblackdynamiteOr directly for the GitLab repository:pipinstallhttps://gitlab.com/ganciaux/blackdynamite.gitGetting the sourcesYou can clone the GIT repository:gitclonehttps://gitlab.com/ganciaux/blackdynamite.gitInstalling completionTo benefit the autocompletion forBlackDynamiteyou need to
activate the global completion as described in the argcomplete website:Howto activate global completion.Introduction and philosophyBlackdynamiteis merely a tool to help
managing parametric studies. In details it comprises:Launching a program repeatedly with varying parameters, to explore the
chosen parametric space.Collect and sort results ofSmall sizesbenefiting
from the power of modern databases.Analyze the results by making requests to the associated databases.Launchingis made simple by allowing any executable
to be launched. The set of directories will be generated and managed
by BlackDynamite to prevent errors. Requests of any kind will then
be made to the underlying database through friendly commands of BlackDynamite.Collectingthe results will be possible thanks to the Blackdynamite C/C++ and python
API which will let you send results directly to the database and thus automatically sort them. This is extremely useful. However heavy data such as Paraview files or any other kind of data should not be pushed to the database for obvious performance issues.Analysisof the results can be made easy thanks to Blackdynamite which
can retrieve data information in the form of Numpy array to be used, analyzed or plotted
thanks to the powerful and vast Python libraries such as Matplotlib and Scipy.The construction of aBlackDynamiteparametric study follows these steps:Describing the parametric spaceCreating jobs (specific points in the parametric space)Creating runs (instances of the jobs)Launching runsIntrumenting the simulation to send resultsAnalyzing the resultsSetting up a parametric studyThe parametrization of a study is done in a YAML file, labelledbd.yaml.
It contains the information of the parametric space, spanned exploration and
configuration of your simulations. An example of a working study is provided in
theexampledirectory.A study description starts with a provided name in the YAML format:---study:bd_studyChoose the parameters of the studyJob descriptionThe first thing to do is to list all the parameters characterizing
a specific case. These parameters can
be of simple scalar types (e.g. string, integers, floats), however no
vectorial quantity can be considered as an input parameter.
It describes theJobpattern of the study.
This must be defined in a section in the thebd.yamlfile.
For instance a three parameter space can be declared as:job:param1:floatparam2:floatparam3:strBy default there is one more entry to every job: its uniqueid.Run descriptionAside from the jobs, a run will represent a particular realisation (computation)
of a job. For instance, the run will contain information of the machine
it was run on, the executable version, or the number of processors employed.
For instance creating the run pattern can be done with:run:compiler:strBy default there are entries created for the user:id: the id of the runmachine_name: the name of the machine where the run must be executednproc: number of processors used to perform the computation (default: 1)run_path: the directory where the run will be created and launchedjob_id (integer): the ID of the running jobstate (str): the current state of the run (CREATED,FINISHED,ERROR)run_name (string): the name of the run (usually a name is given to a collection of runs, at creation)start_time (datetime): time when the run startedlast_step_time (datetime): last time a quantity was pushed to the databaseCreate the databaseThen you have to request for the creation of the database which can be done
with a simple command:canYouDigItinit--truerunAs mentioned, all BlackDynamite
scripts inherit from the parsing system. So that when needing to launch
one of these codes, you can always claim for the valid keywords:canYouDigItinit--help

usage:canYouDigIt[--studySTUDY][--hostHOST][--portPORT][--userUSER][--passwordPASSWORD][--bdconfBDCONF][--truerun][--constraintsCONSTRAINTS][--binary_operatorBINARY_OPERATOR][--list_parameters][--yes][--logging][--help]createDB

General:--loggingActivatethefileloggingsystem(default:False)--helpActivatethefileloggingsystem(default:False)BDParser:--studySTUDYSpecifythestudyfromtheBlackDynamitedatabase.ThisreferstotheschemasinPostgreSQLlanguage(default:None)--hostHOSTSpecifydatabaseserveraddress(default:None)--portPORTSpecifydatabaseserverport(default:None)--userUSERSpecifyusernametoconnecttodatabaseserver(default:tarantino)--passwordPASSWORDProvidesthepassword(default:None)--bdconfBDCONFPathtoaBlackDynamitefile(*.bd)configuringcurrentoptions(default:None)--truerunSetthisflagifyouwanttotrulyperformtheactiononbase.Ifnotsetallactionaremainlydryrun(default:False)--constraintsCONSTRAINTSThisallowstoconstraintrun/jobselectionsbyproperties(default:None)--binary_operatorBINARY_OPERATORSetthedefaultbinaryoperatortomakerequeststodatabase(default:and)--list_parametersRequesttolistthepossiblejob/runparameters(default:False)--yesAnswerallquestionstoyes(default:False)An important point is that most of the actions are only applied
when thetruerunflag is set.Creating the jobsThe goal of the parametric study is to explore a subpart
of the parametric space. We need to create jobs that are
the points to explore.We need to describe the desired set of jobs, to be explored.
This is done in the YAML file describing the study, under the
sectionjob_space. For instance it could be:job_space:param1:10param2:[3.14,1.,2.]param3:'toto'The actual insertion of jobs can be done with the command:canYouDigItjobscreate--truerunYou can control the created jobs with:canYouDigItjobsinfoIn the case of ourexample, 3 jobs should be created
as a range of values for the second parameter was provided.Creating the runsAt this point the jobs are in the database. You need to create runs
that will precise the conditions of the realization of the jobs,
by giving the value of the run space.This is specified in the YAML file under the section run_space. For instance
with:run_space:compiler:'gcc'The default parameters for runs will then be automatically
included in the parameters for the not provided ones (e.g.state).A run now specify what action to perform to realize the job.
Therefore, one script must be provided as an entry point to each run execution.
This will be given in the YAML file as theexec_file. For instance from theexamplea bash script is the entry point and provided as follows:exec_file:launch.shUsually, an end-user has a script(s) and configuration files
that he wishes to link to the run.
This can be done with:config_files:-config.txt-script.pyFinally, we have to create Run objects and attach them to jobs,
which is done with the command:canYouDigItrunscreate--truerunAfter that, all created runs should be present in the database in the stateCREATED, ready to be launched. This can be controled with the command:canYouDigItrunsinfoInstrumentingTextsimulation files (e.g. a bash script)BlackDynamitewill replace specific text marks in the registered files
with the values from the job and run particular point. A precise syntax is
expected forBlackDynamiteto recognize a replacement to be performed.For instance:echo __BLACKDYNAMITE__param1__shall be replaced by the value ofparam1parameter at the run creation.As an additional example, the scriptlaunch.shtaken
from theexamplehas lines such as:echo 'here is the job'
echo __BLACKDYNAMITE__id__
echo __BLACKDYNAMITE__param1__
echo __BLACKDYNAMITE__param2__
echo __BLACKDYNAMITE__param3__Instrumenting aPythonsimulationIn a python program, one can benefit from the possibilities ofPythonto
get a handle object on the current job and run.
This will also allow to push produced data to the database.
This is done by the simplified commands:# import BlackDynamiteimportBlackDynamiteasBD# get the run from the current scopemyrun,myjob=BD.getRunFromScript()In order to have time entries for runs, the functionsstartandfinishneed to be called:myrun.start()...# your code...myrun.finish()Finally, to push data directly to the database, one can usepushVectorQuantityand/orpushScalarQuantity, attached to
meaurablequantities:# pushing vector types (numpy)myrun.pushVectorQuantity(vector,step,"quantity_id")# pushing scalar typesmyrun.pushScalarQuantity(scalar,step,"quantity_id")Executing the runsOnce the runs are created, they can be launched with a command likecanYouDigIt runs launch --truerunDuring and after the run the status can be controlled, once again, with:canYouDigItrunsinfoFor detailed information on a specific run:canYouDigItrunsinfo--run_idRUN_ID_NUMBERin order to be placed in the context of a specific run:canYouDigItrunsinfo--run_idRUN_ID_NUMBER--enterto execute a specific commandcanYouDigItrunsinfo--execCOMMANDand applied for a specific run:canYouDigItrunsinfo--run_idRUN_ID_NUMBER--execCOMMANDManipulating the databaseSelecting jobs and runsAll the previous commands may be applied to a subset of runs/jobs.
In order to select them one should place constraints, provided by
the option--constraint.
For instance, listing the runs constraining parameters
labeledparam1andparam2could be made with:canYouDigItrunsinfo--constraint'param1 > 1, param2 = 2'In the exceptional case where parameters of jobs and runs would bear the same name
(you should avoid to do that), one can disambiguate the situation with:canYouDigItrunsinfo--constraint'jobs.id > 1, runs.id = 2'Cleaning RunsSometimes it can be necessary to re-launch a set of runs. Sometimes it
can be necessary to delete a run. In order to reset some runs,
making them ready to relaunch, one should use the following:canYouDigItrunsclean--constraint'jobs.id > 1, runs.id = 2'--truerunTo completely delete them:canYouDigItrunsclean--constraint'jobs.id > 1, runs.id = 2'--delete--truerunAltering runsRarely it can be necessary to manually change a set of runs. For instance changing the state of a set of runs can be done with:canYouDigItrunsupdate--truerunstate=totoPlotting resultsFor starting the exploration of the collected data, and thus producing graphs,
theplotcommand can be employed. While tunable, it cannot produce any type of graphs. However, for quick exploration of the data, one could do:canYouDigItrunsplot--quantityekin--legend"%r.id"--markeroExporting the results (to zip file)Under ConstructionFecthing the resultsUnder construction...