Package: lab-notebook
Lab NotebookResearchers in computer science often need to compare results between different versions of a process.Lab Notebookhelps document, track, and organize process these kinds of runs.
This is essential for reproducibility and helps researchers figure out what changes in code led to different outcomes.
The goals ofLab Notebookare reproducibility, modularity, and organization.
Specifically,Lab Notebookprovides the following functionality:Maintain metadata about each run, including a description, a timestamp, and a git commit.Automatically set up runs, building flags and directories with unique name corresponding to each run and launching runs in tmux.Organize runs into hierarchical categories.Synchronize runs with directories, so that directories are moved and deleted when runs are moved and deleted.InstallationThe only external dependencies of this tool aretmuxandgit. After that,pip installlab-notebook.ConfigurationThe program will default to any arguments specified in.runsrc.
The user can always override the.runsrcfile with command-line arguments.
For descriptions of arguments, useruns-horruns [command]-hThe program searches for the.runsrcfile in ancestors (inclusive) of the current working directory.
If the program does not find a.runsrcfile, it will create one with default values in the current working directory.
The user can use two keyword in the.runsrc:<path>will be replaced by thepathto the run. Paths look just like ordinary file paths (/-delimited).<name>will be replaced by the head ofpath.Also users can interpolate strings from other sections of.runsrcusing the syntax${section:value}.
For more details seeconfigparser ExtendedInterpolation.Here is an example.runsrcfile:[main]root=/Users/ethan/demo-lab-notebook/.runsdb_path=/Users/ethan/demo-lab-notebook/runs.pkldir_names=tensorboardprefix=Source ~/virtualenvs/demo-lab-notebook/bin/activate;nice[flags]--log-dir=${main:root}/tensorboard/<path>[new]description=demo lab-notebookThis will pass the flag--logdir=/Users/ethan/baselines/.runs/tensorboard/<path>to any program launched withrun, where<path>will be replaced by thepathargument given by the user.runs-gitThis is a simple wrapper aroundgitthat substitutes+your-pathwithruns lookup commityour-path.
For example, to see changes since when you launchedyour-run:runs-git diff +your-runIf you want to live on the wild side, usedirenvto aliasgittoruns-gitwhen you
are in your project directory.Example UsageSetup environment:mkdir ~/lab-notebook-demo/ && cd ~/lab-notebook-demo
wget https://raw.githubusercontent.com/tensorflow/tensorflow/master/tensorflow/examples/tutorials/mnist/mnist_with_summaries.py
pip install tensorflow lab-notebook
git init
echo 'runs.pkl .runs .runsrc' > .gitignore
git add -A
git commit -am initCreate a new run. The run will be launched intmux:runs new train 'python mnist_with_summaries.py' --description='demo new command'Check out your run:tmux attach -t trainReproduce your run:runs reproduce train
runs reproduce --no-overwrite trainTry modifying the.runsrcfile to look like the example in theConfigurationsection with appropriate changes for your system.
Then create a new run:runs new subdir/train 'python mnist_with_summaries.py' --description='demo categorization'Get an overview of what runs are in the database:runs ls
runs ls 'tra*'
runs ls --show-attrsQuery information about current runs:runs lookup description train
runs lookup commit trainruns-git: avoid typingruns lookup commit <path>all the time:echo '# Hello' >> mnist_with_summaries.py
runs-git diff +trainOrganize runsruns mv train subdir/train2
runs ls
tree .runs  # note that directories are synchronized with database entries
runs mv subdir archive
runs lsDelete runsruns rm archive/train
runs killallSubcommandsFor an overview of subcommands, runruns -hFor detailed descriptions of each subcommand and its arguments, runruns <subcommand> -hTab autocompletionIf you are using Zsh, simpy copy the_runsto some place on yourfpath.
Then pressing tab will prompt you with the names of runs currently in your
databaseWhy not just use git?If processes are long-running, it is hard to know which commit a given run corresponds to.Commit statements are really meant to describechangesto software, notruns. A description of a change may not actually tell you very much about the motivation for a software run.Not all commits will correspond to runs, so you will need to fish through a large number of commits to find those that correspond to runs.Often processes depend on specific file-structures (e.g. a logging directory). Setting up and removing these directories by hand is time-consuming and error-prone.Commits cannot be organized hierarchically or categorized after their creation.