Package: bamboost
BAMBOOSTBamboost is a Python library built for datamanagement using
the HDF5 file format.
bamboost stands for alightweightshelf which willboostyour efficiency and which
will totally break if you load it heavily. Just kidding, bamboo can fully carry pandas.🐼🐼🐼🐼DocumentationInstallationInstall the latest release from the Package repository:pip install bamboost:warning: If you're system runs into problems installingmpi4py, make sure python header files are installed. Quickly google what you need (something likepython3-dev,libpython3.8-dev, etc.).Install the package in editable mode for more flexibility, $e.g.$ if you plan to make changes yourself:git clone git@gitlab.com:cmbm-ethz/bamboost.git
cd bamboost
pip install -e .:warning: The option-einstalls a project in editable mode from a local path. This way,
you won't need to reinstall when pulling a new version or changing something in the
package.h5py with parallel supportFor mpi support,h5pymust be installed with parallel support. Otherwise, eachp
process writes one after the other which takes forever. The default installation on
Euler is not enough.It's simple, do the following:export CC=mpicc
export HDF5_MPI="ON"
pip install --force-reinstall --no-deps --no-binary=h5py h5pyRequirementspython > 3.7(if you're version is too low, it's very likely only because of typehints.
Please report and we can remove/change it)bamboostdepends on the following packages:numpypandash5pympi4pyUsageManagerThe main object ofbamboostis theManager. It manages the database located in the directory
specified during construction. It can display the parametric space, create new simulations, remove simulations
select a specific simulation based on it'suidor on conditions of it's parameters.
Every database that is created is assigned a unique identifier (UID).frombamboostimportManagerdb=Manager('path/to/db')pandas.DataFrameis used to display the database. The dataframe is convenient and fast to filter or sort your entries:db.dfAn entry (from now on calledsimulation) within a database can be viewed, retrieved and modified with theSimulationobject.
To get theSimulationobject, access it with it's identifier or location (index) in the dataframe:sim=db['uid']sim=db[index]sim=db.sim('uid')All simulations can be returned as a (sorted) list. The argumentselectcan be used to
filter the simulations.sims=db.sims()# returns allsims=db.sims(select=(db.df.eps==1))# returns all where eps is 1sims=db.sims(sort='parameter1',reverse=False)# returns all, sorrted by parameter1:warning: Note that this creates objects for every simulation and the sorting is not optimized. Using pandas to select and sort is much faster. Check their documentation for how to manipulate pandas dataframes.Database indexEvery database created will be assigned a unique identifier (UID).
The database path is stored with the UID in an index maintained at~.config/bamboostin your home directory. If it is not known,bamboostwill try to find it on your disk (you can add paths to search in~.config/bamboost/known_paths.json).
You can obtain a Manager object of any database from anywhere with it's UID. In notebooks, key completion will show you all known databases:db=Manager.fromUID['UID']The unique id makes refering to data safe. The full identifier of a simulation is considered to be'(database id):(simulation id)'. It is encouraged to use the identifiers (instead of the path) to link from one simulation to a different one.# add a link to a different simulation (e.g. the mesh)sim.links['mesh_to_use']='DATABASE-ID:simulation-id'# the full id of a simulation is accessible as suchuid=sim.get_full_uid()Write dataYou can usebamboostto write simulation or experimental data.
Use theManagerto create a new simulation (or access an existing one).
Say you have (or want to create) a database atdata_path.
The code samples below shows the main functionality.frombamboostimportManagerdb=Manager(data_path)params={...}# dictionary of parameters (can have nested dictionaries)writer=db.create_simulation(parameters=params)writer.copy_file('path/to/file/')# copy a file which is needed to the database folder (e.g. executable, module list, etc.)writer.change_note('This run is part of a series in which i investigate the effect of worms on apples')# Use context manager (with block) and the file will be tagged 'running', 'finished', 'failed' automaticallywithwriter:writer.add_metadata()# adds time and other metadatawriter.add_mesh(coordinates,connectivity)# Add a mesh, default mesh is named 'mesh'.writer.add_mesh(coordinates,connectivity,mesh_name='interface')# add a second mesh for e.g. the interface# loop through your time data and writefortintimes:writer.add_field('field_data_1',array,time=t)writer.add_field('field_data_2',array,time=t,mesh='interface')writer.add_global_field('kinetic_energy',some_number)writer.finish_step()# this increases the step counterIf you have an existing dataset, $e.g.$ because you created the simulation before and it holds the input parameters or similar. Do the following: You will need to pass the path and the uid to the script (best useargparse).frombamboostimportSimulationWriterwithSimulationWriter(path,uid)aswriter:# Do anythingUserdata (Data not related to time and/or space)The above functionality should be used for ordered data, such as timeseries of spatial data related to a mesh.
For anything else, there is theuserdatacategory. You can use it to store (almost) anything in the simulation file structured how you would like it. This is also useful to store computed values during postprocessing or plotting.
Internally, Userdata is an object handling a specific group ('/userdata') of the hdf5 file. To show the content of the group, display the object:sim.userdataYou can create a subgroup, which will return a self-similar object for the new group ($e.g.$ '/userdata/plots'):plot_grp=sim.userdata.require_group('plots')Writing something to the file (group) is as easy as:sim.userdata['avg_T']=34.56256sim.userdata['traction_profile']=np.array([...])And reading:# read avg_Tsim.userdata['avg_T']# read dataset traction_profilesim.userdata['traction_profile']# note that this returns an object Dataset. To actually read the array, you will need to slice itsim.userdata['traction_profile'][:]Read dataThe key purpose is convenient access to data. I recommend an interactive session (notebooks).Display databasefrombamboostimportManagerdb=Manager(data_path)To display the database with its parametric space simply inputdb.dfSelect a simulation of your dataset.simwill be aSimulationReaderobject.sim=db[index]sim=db[uid]sims=db.sims((db.df.param1==2)&(db.df.param2>0),sort='param2')# will return list of all matching, sorted by param2Show data stored:Display content of the data, userdata, globals groups:sim.data
sim.userdata
sim.globalsThis displays the stored fields and its sizes.sim.data.infoAccess a mesh:Directly access a tuple where [0] is the coordinates, [1] is the connectivity.coords,conn=sim.mesh# default meshcoords,conn=sim.get_mesh(mesh_name=...)You can get a mesh object the following way.mesh1=sim.meshes['mesh1']mesh1.coordinates# gives coordinatesmesh1.connectivity# gives connectivitymesh1.get_tuple()# gives both the aboveAccess field data:sim.dataacts as an accessor for all field data.field1=sim.data['field1']field1[:],field1[0,:]# slice the dataset and you get numpy arrays (time, *spatial)field1.at_step(-1)# similar for access of one stepfield1.mesh# returns the linked mesh object (see above)field1.msh# returns a tuple of the mesh (coordinates, connectivity)field1.coordinates,field1.connectivity# direct access to linked mesh' coords and conn arraysfield1.times# returns timesteps of datafield1.shape# shape of datafield1.dtype# data type of dataAccess global data:sim.globalskinetic_energy=sim.globals.kinetic_energyOpen file:All methods internally open the HDF5 file and make sure that it is closed again. Sometimes
it's useful to keep the file open (i.e. to directly change something in the file manually).
To do so, you are encouraged to use the following.:warning: Do not open the file in write mode ('w') as this truncates the file.withsim.open(mode='r+')asfile:# do anything# in here, you can still use all functions of the bamboost, the functions will not close# the file in the case you manually opened the file...Job managementYou can usebamboostto create euler jobs, and to submit them.frombamboostimportManagerdb=Manager(data_path)params={...}# dictionary of parameters (can have nested dictionaries)sim=db.create_simulation(parameters=params)sim.copy_file('path/to/postprocess.py')# copy a file which is needed to the database folder (e.g. executable, module list, etc.)sim.copy_file('path/to/cpp_script')sim.change_note('This run is part of a series in which i investigate the effect of worms on apples')# commands to execute in batch jobcommands=[]commands.append('./cpp_script')commands.append(f'mpirun python{os.path.join(sim.path,'postprocess.py')}')# e.g. to write the output to the database from cpp outputsim.create_batch_script(commands,ntasks=4,time=...,mem_per_cpu=...,euler=True)sim.submit()# submits the job using slurm (works only in jupyterhub sessions on Euler)To be continued...Feature requests / IssuesPlease open issues on gitlab:cmbm/bamboost