Package: nbx
NBX: Notebook Experiments for OpenMindThis module enables you to quickly convert your jupyter notebook into a bundle of files that can be run on BCS'OpenMind.Getting startedPrerequisitesInstall the package:pip install nbxGet a singulartiy imageYou'll need an image that has the package installed (there are ways around that, but I am keeping it simple at the moment). Here's how you can build an image:module load openmind/singularity/3.2.0
export SINGULARITY_CACHEDIR="/om2/user/{your_user_name}/.singularity"
singularity build pytorch.simg docker://mklukas/pytorchEnvironment variablesFor the modules to work you have to set the environment variablesom,omx,omsimg, andomid:om: your login toOpenMind.You need to enable logging intoOpenMindusing public key authentication. That means the commandssh $omshould log you in whithout asking for a password. (googling for "ssh public key authentication" will provide you with a recipe likethis)omx: path to the folder wherenbxbundles are stored. This path will automatically be added to your python path. Any modules that are not part of your bundle'ssrc/folder or are included in your singularity container should go here.omsimg: path to the folder containing your singularity imagesomid: your Open Mind usernameMac users can adapt and copy the following lines to their.bash_profilefileexport om={your_user_name}@openmind7.mit.edu
export omid={your_user_name}
export omx=/om2/user/{your_user_name}/nbx-experiments
export omsimg=/om2/user/{your_user_name}/simgUsagePut your python scripts that you wrote for this experiment in a folder calledsrc. The folder will be copied to the bundle so the scripts are available onOpenMindas well.#nbx: Each cell that contains a#nbxtag in itsfirstline will be considered part of the experiment.#xarg: Putting#xargabove a variable declaration makes this variableexplicit, it will become an argument of the experiment function. Any iterable to the right of the variable declaration,separated by a semicolon, will be considered the domain that will be swept during the parameter sweep.Each nbx-experiment has to declare the variablestask_idandresults_dir. Thetask idwill be set by thewrapperscript and enumerates the configurations of the parameter space. The latter variable will also be set by thewrapperscript, it will be replaced by the folder automatically created for a specific parameter configuration.ExampleExperimentIn every experiment we need to indicate which cells are part of it (using the#nbxflag), and need to specify these two arguments:task_idresults_dir%matplotlibinline%load_extautoreload%autoreload2#nbx#xargtask_id=0#xargresults_dir="."This cell will be part of the experiment#nbx#xargx=0;range(5)#xargy=0;[0,1,2,4]z=0;# ...This cell will also be part of the experiment. The output will be written to a log file in theiofolder that will automatically be created.#nbxprint("my results:",x,y,z)my results: 0 0 0Note how we used the variableresults_dir. It will will be replaced by"results/task_id/"; a corresponding folder will automatically be created. It is really just a hook so we can manipulate it behind the scenes.#nbxwithopen(f"{results_dir}/your_file.txt","w")asf:f.write("I will be written to: example_nbx_bundle/results/task_id/your_file.txt")f.write(f"\n{task_id}")Creating and running an NBX bundleTo run the experiment on OM we have to create a bundle that we can interact with...fromnbx.omimportNbxBundlebundle=NbxBundle(nbname="index.ipynb",# the name of the notebook to use as expname="example_bundle",# name of the bundlelinting=False,# enable basic lintingtime=[0,20],# comp time [hours, minutes]ntasks=4,# requested comp nodesstep=50,# parallel jobs (compare bundle/run.sh)max_arr=10,# maximum number of queued jobs on OM is 1000mail_user="me@somewhere.com",# notification emailsimg="pytorch.simg")# singulrity img on OM in $omsimg** nbx bundle created **
Path:
    example_bundle_nbx

Source nb:
    index.ipynb

Parameters (#configs 20):
    * x = range(5)
    * y = [0,1,2,4]
      task_id = 0
      results_dir = "."

Instructions:
    Copy to remote, run the bash script, and pull the results
    - `bundle.push()` or `scp -r example_bundle_nbx $om:$omx`
    - `bundle.run()` or `ssh $om sbatch -D $omx/example_bundle_nbx $omx/example_bundle_nbx/run.sh`
    - `bundle.pull_results()` or `scp -r $om:$omx/example_bundle_nbx/results ./results`!lsexample_bundle_nbx/__init__.py   experiment.py job.sh        wrapper.py
[34m__pycache__[m[m   [34mio[m[m            run.shfromexample_bundle_nbx.experimentimportsweep_paramsassweepprint(len(sweep))print(sweep[0])print(sweep[1])print(sweep[4])12
{'x': 0, 'y': 0}
{'x': 0, 'y': 1}
{'x': 1, 'y': 0}bundle.push()bundle.run()bundle.status()bundle.pull_results()The results are now in the local folder:!lsexample_bundle_nbx