Package: overboard
OverBoardOverBoard is a lightweight yet powerful dashboard to monitor your experiments.Loadremote experiments in real-timethroughSSH/FTPandothers.Custom visualisations (tensors and any custom plot with familiarMatPlotLibsyntax).A sortable table ofhyper-parameterswith Python-syntax filtering.Plot percentile intervals (shaded plots), custom X/Y axes, and hyper-parameters (bubble plots).InstallationYou can install the dependencies with:With Conda:conda install pyqt=5.12 pyqtgraph=0.11 -c conda-forgeWith pip:pip install pyqt5==5.12 pyqtgraph==0.11Finally, OverBoard itself can be installed with:pip install fs==2.4 overboard(Conda seems to be too strict when installingPyFileSystem/fs, so pip should be preferred.)Optional:pip install fs.sshfsto support remote files through SSH.PyOpenGL 3.1 (either through conda or pip) if you intend to use custom 3D plots withPyQtGraph.MatPlotLib if you intend to use it for custom plots.Installation - logger onlyYour scripts can log data without installing the full GUI and its dependencies (so your remote GPU cluster does not need PyQt at all).Just use:pip install overboard_loggerAnd remember to importoverboard_loggerinstead ofoverboardin your scripts.UsageMain interface:python3 -m overboard <logs-directory>Logging experiments is simple:fromoverboardimportLoggerwithLogger('./logs')aslogger:foriterationinrange(100):logger.append({'loss':0,'error':0})You can also pass in ametakeyword argument, which can be adictwith hyper-parameters names and values (or other meta-data), to help organize your experiments. These will be displayed in a handy table, which supports sorting and filtering. Themetadata can also be anargparse.Namespace, which is useful if your hyper-parameters are command-line arguments parsed withargparse.By default a unique folder (using the current timestamp) is created for the logs. For full documentation on initialization arguments and other methods, typepydoc overboardon the command-line (Python built-in doc viewer).You can also check theexamplesdirectory:examples/basic.py: A minimal example. Generates some test logs.examples/mnist.py: The mandatory MNIST example. Also shows a custom MatPlotLib plot (a confusion matrix).examples/example_2d.py: Example custom 2D plot, with PyQtGraph (faster than MatPlotLib).examples/example_3d.py: Example custom 3D plot, with PyQtGraph. Requires PyOpenGL.A note about importing: You can either import theLoggerclass fromoverboardor fromoverboard_logger. If you installed the "logger only" version as described above (no dependencies), then you can only import fromoverboard_logger.Remote experimentsIt's as simple as:python -m overboard ssh://username:password@hostname/path-to-experiments...replacing your SSH user name, password, host name/server, and the directory where OverBoard should look for experiments. The default port is 22, which you can override with :23 or another port number after the host name.There are anumber of other settings, which you can append after the path. For example, append?keepalive=60&config_path=~/.ssh/configto set the keepalive packets interval to 60 seconds, and specify an OpenSSH configuration file path.Other remote file systemsare available by usingappropriate prefixes(e.g.ftp://).Interface tips & tricksClick a plot title to hide it (moving it to the bottom) or to show it again.Click a plot line to select that experiment in the table, and vice-versa.Use the mouse wheel to zoom and drag to pan in a plot or custom visualization.Click the "A" button in the bottom-left of a plot to reset the zoom/pan.Choose an hyper-parameter as the "X axis" in the sidebar to compare different hyper-parameters graphically (bubble plot). Click a bubble to highlight that experiment. If the hyper-parameter is numerical but has wildly different orders of magnitude, check "Treat X as categorical". By default, each bubble's Y coordinate (e.g. accuracy) is taken from the last iteration; in "Scalar display" you can choose the maximum or minimum across iterations.If you have multiple runs with different random seeds (stored as hyper-parameter "seed" for example), select "Merge: seed" in the sidebar to merge them intoshaded plots(you can then select whether to show the mean, median, range or standard deviations).You cansplitall experiments into different plots by selected "Panels: One per run" in the sidebar, and selecting a metric to plot in "Y axis". Similarly, they can be split by a hyper-parameter value (e.g. a different algorithm in each panel, but each has multiple plots/runs).AuthorJoão Henriques,Visual Geometry Group (VGG), University of Oxford