Package: tf-run-manager
Machine learning engineers often run multiple versions of an algorithm concurrently. However, this can make keeping track of and reproducing runs difficult. This simple utility solves this problem by maintaining a database in human-readable YAML formal that tracksA unique name assigned to each run.A description of each run.The exact command used for the run.The date and time of the run.The most recent commit before the run.InstallationThe only external prerequisites of this tool aretmuxandgit. After that,pip installrun-manager.Important paths and filesWhen you runruns new, the utility automatically creates the following directory structure:<Runs Directory>/
    <Runs Database>
    checkpoints/
    tensorboard/<Run Name>/Runs DatabaseYAML file that stores historical information about Tensorflow runs.Run NameThis is a unique value that you assign to each run. Therunssection explains how the program deals with collisions.checkpointsdirectoryDirectory where model checkpoints are saved. Used intf.train.Saver().save(sess,<checkpointsdirectory>/<RunName>.ckpt).tensorboarddirectoryDirectory where events are saved. Used intf.summary.FileWriter(<tensorboarddirectory>/<RunName>/).ConfigurationRuns can be extensively configured using command-line arguments, but the following values can also be configured in a.runsrcfile:namedefaultdescriptionruns-dir.runs/The name to use for your Runs Directory.db-filename.runs.ymlThe name that you choose to save your runs database with.tb-dir-flag--tb-dirThe flag that gets passed to your program that specifies<tensorboarddirectory>/<RunName>/. IfNone, no flag will be passed to your program.save-path-flag--save-pathThe flag that gets passed to your program that specifies<checkpointsdirectory>/<RunName>. IfNone, no flag will be passed to your program.column-width30The default column width for theruns tablecommand.virtualenv-pathNoneThe path to your virtual environment directory, if you’re using one. Used in the following command:Source<virtualenv-path>/bin/activate.extra-flags[]Flag, value pairs for extra, custom flags. The strings<runs-dir>and<run-name>will get replaced with the appropriate value.The program expects to find the.runsrcin a parent of the current working directory. Unless specified otherwise, the.runs/directory will be adjacent to the.runsrcin the file structure.Here is an example.runsrcfile:runs-dir:.lstm-runs/db-filename:lstm-runs.ymltb-dir-flag:Nonesave-path-flag:-scolumn-width:10virtualenv-path:/home/ethan/virtualenvs/baselines/extra-flags:-[--goal-log-dir,<runs-dir>/goal-logs/<run-name>.log]AssumptionsThis program tries to assume as little about your program as possible, while providing useful functionality. These assumptions are as follows:You call therunscommand from a directory whose parent contains the runs directory.Your program lives in a Git repository.The Git working tree is not dirty (if it is, the program will throw an informative error).Your program accepts a--tb-dirflag, which your program uses intf.train.Saver().save(sess,<tf-dir>), and a--save-pathflag, which your program uses intf.train.Saver().restore(sess,<save-path>). If your flags are different and you don’t feel like changing them, you can specify the new flag names using command-line arguments (--tb-dir-flagand--save-path-flag) or in your.runsrc(see theConfigurationsection for more info). If you don’t want to pass either flag to your program, set--tb-dir-flagor--save-path-flag(or the associated values in your.runsrc) toNone.SubcommandsFor detailed descriptions of each subcommand and its arguments, runruns <subcommand> -hnewStart a new run and build the file structure (seeImportant paths and files).It will add an entry to the database keyed by name, with the following values:commandcommitdatetimedescriptionhostFinally, it will execute the command intmux.runs new 'run-name' 'python main.py' --description='Description of program'Note:the--tb-dirand--save-pathflags will be automatically
appended to this command argument, so do not include them in the<command>argument.deleteDelete all runs matching pattern. This command also deletes associated tensorboard and checkpoint files.❯ runs delete "continuous.*"
Delete the following runs?
continuous0
continuous1
continuous21509805012
continuous2
continuous11509804959
continuous3
continuous31509805040listList all runs matching pattern.❯ runs list --pattern="continuous.*"
continuous21509805012
continuous0
continuous11509804959
continuous31509805040
continuous1
continuous2
continuous3tableDisplay entries in run-database in table form.❯ runs table
name                           command                            commit                             datetime                    description                          host
-----------------------------  ---------------------------------  ---------------------------------  --------------------------  ---------------------------------  ------
continuous2                    CUDA_VISIBLE_DEVICES=1 python ...  90c0ad704e54d5152d897a4e978cc7...  2017-11-03T13:46:48.633364  Run multiple runs to test stoc...    rldl3
continuous3                    CUDA_VISIBLE_DEVICES=1 python ...  90c0ad704e54d5152d897a4e978cc7...  2017-11-03T13:47:09.951233  Run multiple runs to test stoc...    _
continuous1                    CUDA_VISIBLE_DEVICES=1 python ...  90c0ad704e54d5152d897a4e978cc7...  2017-11-03T13:42:39.879031  Run multiple runs to test stoc...    _
house-cnn-no-current-pos       python train.py --timesteps-pe...  9fb9b5a                            2017-10-28T18:07:44.246089  This is the refactored CNN on ...    _
room-with-original-cnn         python run_custom.py --timeste...  8a5e1c2                            2017-10-28T17:09:49.971061  Test original cnn on room.mjcf       _
continuous11509804959          CUDA_VISIBLE_DEVICES=1 python ...  90c0ad704e54d5152d897a4e978cc7...  2017-11-04T10:15:59.373633  Run multiple runs to test stoc...    _
continuous31509805040          CUDA_VISIBLE_DEVICES=1 python ...  90c0ad704e54d5152d897a4e978cc7...  2017-11-04T10:17:20.286275  Run multiple runs to test stoc...    rldl4
room-cnn-no-current-pos        python train.py --timesteps-pe...  2873fbf                            2017-10-28T18:08:10.615461  This is the refactored CNN on ...    rldl4
continuous21509805012          CUDA_VISIBLE_DEVICES=1 python ...  90c0ad704e54d5152d897a4e978cc7...  2017-11-04T10:16:52.129656  Run multiple runs to test stoc...    _To filter by regex, use--patternflag.lookupLookup specific value associated with database entry.❯ runs lookup continuous0 commit
da6030dd973c810c330d9635eb8d9c2105bdfe2freproducePrint out commands for reproducing run.❯ runs reproduce continuous0
To reproduce:
 git checkout da6030dd973c810c330d9635eb8d9c2105bdfe2f
 runs new continuous0 'python run_custom.py --timesteps-per-batch=2048 --continuous-actions --neg-reward --use-cnn' --description='None'