Package: lavd
Log and Visualise Data (Lavd)Tracking the progress of Machine Learning (ML) experiments is essential for
deciding what model works well with the available data. Using some kind of
metric, for example Intersection over Union (IoU) for object detection, gives
a good idea of the quality of the model, but additionally, it is important to
also see a particular example to get a better intuition. This, and more can be
logged and visualised in an interactive manner, including:ScalarsImagesTextLog filesMarkdown documentsCLI commands (how to run the experiment with the same parameters)...and more to come.Table of ContentsTable of ContentsGetting StartedLogging DataScalarsImagesTextMarkdownCommand Line OptionsSummaryPrintingSpinnerProgress BarSaving a PyTorch ModelSaving any ObjectDisabling the LoggerData LayoutComparison to TensorBoardKnown IssuesInotify watch limit reachedGetting StartedInstalling with pip:pipinstall--user--upgradelavdOnce installed, the server can be started by runninglavdwith the directory
containing the (logged) data:lavdpath/to/logsOr if you prefer to run it as a module instead of using the executable (both
versions do exactly the same):python-mlavd.serverpath/to/logsLogging DataLavdincludes a logger that can be used to easily log all the desired data.importlavd# Logs everything to log/some-experiment-name/# If no name is giving it the current date and time are used as a namelogger=lavd.Logger("some-experiment-name")ScalarsScalars will be plotted and unlike other categories, the have to be assigned to
a step.logger.log_scalar(0.8,"accuracy",step=1)logger.log_scalar(0.6,"accuracy",step=2)logger.log_scalar(0.3,"accuracy",step=3)logger.log_scalar(0.1,"easy/accuracy",step=7)logger.log_scalar(0.05,"easy/accuracy",step=14)ImagesLogging images with optional bounding boxes that for interactive visualisation.# Saves image to: log/some-experiment-name/0001/bird.pnglogger.log_image(image,"bird",step=1)# Saves image to: log/some-experiment-name/0001/other/bird.pnglogger.log_image(image,"other/bird",step=1)# No step saves it to: log/some-experiment-name/other/bird.pnglogger.log_image(image,"other/bird")# With bounding boxes, each box is given by 2 points: top-left and bottom-right.boxes=[{"xStart":100,"yStart":100,"xEnd":150,"yEnd":200,"className":"bird",# Optional"probability":0.4,# Optional},# Another bounding box, without class or probability{"xStart":200,"yStart":22,"xEnd":233,"yEnd":80,},]# List of available classesclasses=["bird","orange","background"]# Threshold to count boxes as valid and threfore visiblethreshold=0.2logger.log_image(img,"birds-with-bounding-boxes",step=3,boxes=boxes,classes=class_names,threshold=threshold,)TextText can be logged with an additional expected text, which then shows a diff
between the actual text and the expected, otherwise it's just the text.logger.log_text("The quick brown fox...","famous-sentence",step=1)# With an expected textlogger.log_text("hello world","with-diff",step=2,expected="Hallo Welt")MarkdownMarkdown files are given as Markdown string.logger.log_markdown("# Hello\n\nMore markdown...","some-markdown")logger.log_markdown("# Step 1\\nn## Result\n\nGood","for-step",step=1)Command Line OptionsCommand line arguments that have been used to start the experiment and all
available options for that script can be saved, if you useargparseto parse
the command line options.importargparseparser=argparse.ArgumentParser()# Add all options with: parser.add_argument()# ...args=parser.parse_args()logger.log_command(parser,args)SummaryA summary of the current experiment, including current git commit, diff of the
working tree and possible more information, can be saved as a Markdown document.
There are two parts of the summary that can be configured, short infos that are
presented is an unordered list given as a dictionary, where the key is the label
(can be nested for nested lists), and additional sections at the end of the
document given as a dictionary where the key is the section title and the value
is either the raw string of the content or a list of lines.# Regular summarylogger.log_summary()infos={"size":{"train":1000,"validation":50}}sections={"Additional raw Information":"Some text\nand more","Same but with Lines":["Some text","and more"],}logger.log_summary(infos,sections)PrintingPrint functions act as print() combined with "".format() but also log all the
messages into a log file in addition to printing.#               Message              ...Format argumentslogger.println("{}, message:{num}","Hello",num=1)# => Hello, message: 1# Same but to STDERR instead of STDOUTlogger.eprintln("{}, message:{num}","Hello",num=1)Nicely formatted tables withprint_table:header=["Name","Correct","Total","Accuracy"]rows=[["Train",978,1000,0.978],["Validation",90,100,0.9],["Test",None,50,None],]logger.print_table(header,rows)# | Name       | Correct | Total | Accuracy |# |------------|---------|-------|----------|# | Train      | 978     | 1000  | 0.97800  |# | Validation | 90      | 100   | 0.90000  |# | Test       | -       | 50    | -        |SpinnerA small wrapper aroundHalo, that also logs start and end of the
duration to a log file. Requires a name as the first argument, which will be
used in the log file and as the text next to the spinner, unless thetextargument is supplied. All additional arguments are forwarded toHalo.withlogger.spinner("Loading..."):# Do something that takes a while# Without context manager, but also some arguments for Halospinner=logger.spinner("Manual spinner",spinner="smiley",placement="right")spinner.start()# Do somethingspinner.stop()Progress BarA small wrapper aroundtqdm, that also logs start and end of the
duration to a log file. Requires a name as the first argument, which will be
used in the log file and as the text next to the progress bar, unless thedescargument is supplied. All additional arguments are forwarded totqdm.fordinlogger.progress_bar("Train",data):# Trainings loop# With context manager and manual updatewithlogger.progress_bar("Manual progress bar",total=100)aspbar:whileTrue:# Do something# Update the progress bar by 4 stepspbar.update(4)Saving a PyTorch ModelPyTorchis optional, but must be installed to use this feature.Saves the state dict of a PyTorch model and optionally its current gradients. If
the model is a Multi-GPU/Node model (i.e.nn.DataParallelornn.parallel.DistributedDataParallel) it will automatically be unwrapped to be
compatible with the initial model.model=ResNet101()logger.save_model(model,step=3)distributed_model=nn.parallel.DistributedDataParallel(model,device_ids=[gpu_id],find_unused_parameters=True)# Distributed model and also saving the gradientslogger.save_model(distributed_model,step=4,grads=True)Saving any ObjectPyTorchis optional, but must be installed to use this feature.Saves any object by serialising it withtorch.save.logger.save_obj({"key":99},"a-dict")logger.save_obj((1,2),"some_tuple",step=4)Disabling the LoggerAll logging actions can be disabled, either creating a disabled logger or by
later disabling it. The logging actions will be a no-op instead and all methods
can be used as if they were a regular logger but without having any output.
Methods that produce some useful information without logging anything continue
to function as usual.Disabling the logger is particularly useful when the same script is launched
multiple in multiple processes, but only the main process should create the
logs.# Create a disabled logger# No log directory will be createdlogger=lavd.Logger("some-experiment-name",disabled=True)# Does nothinglogger.println("hello")# Won't show the spinnerwithlogger.spinner("Loading..."):# Do something that takes a while# Create a normal loggerlogger=lavd.Logger("some-experiment-name")# Disable it laterlogger.disable()# Enable it (also works if the logger was initially disabled)logger.enable()Data LayoutThe server picks up any data that is present in the specified log directory that
can be displayed, but certain functionalities require a specific layout or file
format (but still human readable).In the root of log directory (i.e. path given tolavd) the different
experiments are listed by name. Each experiment is in its own directory. Any
other file at the top level will be ignored, since it cannot be associated to
one experiment.Within each experiment directory, there are numbered directories, which acts as
a step (epoch) within the experiment, and all files within that directory
belongs to that specific step. They are usually padded with zeros e.g.0001,
but that is not required, it only helps to show them in the correct order for
tools, such asls, that shows them in lexicographical order. Since0001 == 01 == 1, multiple representations of the same numbers will be in
conflict with each other, and should therefore be avoided.Every other file, including nested files, are used as data based on their file
type and the relative path is used as a category label (minus the extension).
For example, thesome-experiment-name/0001/bird.pngis an image belonging to
thesome-experiment-nameexperiment, of step 1 and with the labelbird.
The same label can contain multiple data, let's say there is also abird.mdfile, so there would be a Markdown document with the labelbird. The same
base name can therefore be used for different files, as long as they are in
a different category, this means that just using a different extension will
create a conflict (e.g.birds.pngandbirds.jpeg) and should also be
avoided.Files not within a step directory are considered global to the experiment
(including nested files) and once in a nested directory, numbered directories
are just regular directories, not specific steps, for examplesome-experiment-name/other/0001/still-global.mdis a global tosome-experiment-namewith the labelother/0001/still-global.There is one exception to the rule above, and that's for commands. Only
command is used per experiment, and it's thecommands.jsonfile, directly
inside the experiment directory, i.e.some-experiment-name/commands.json. Any
other file containing the commands, will not be displayed (just the command, all
other categories are still shown even if they are in a nestedcommand.json)JSON files are special in the sense that they can contain multiple kinds of
data. They are separated within the JSON file by using the appropriate keys.The following directory structure is demonstrates the key ideas:log/
├──some-experiment-name
│├──0001# Step: 1││├──bird.png# Label: bird││├──bird.json# Label: bird││├──bird.md# Label: bird││└──difficult
││└──unlabelled
││└──someimage.png# Label: difficult/unlabelled/some image│├──0002# Step 2││└──bird.png
│├──command.json# Only command that can contain command definitions│├──events.log
│├──other# Nested global││├──0001# Not a step directory!│││└──still-global.md# Label: other/0001/still-global││└──instructions.md# Label: other/instructions│└──summary.md
├──yet-another-experiment
│└──0007│└──bird.png
├──top-level-file.md# ignored│
...Comparison to TensorBoardTensorBoardis one of the most used tools for this purpose and
is undoubtedly incredibly useful. Unfortunately, it also has its drawbacks,
henceLavdwas created with a different set of goals. One of the biggest
annoyances from of TensorBoard is the fact that everything is serialised, which
means that you cannot inspect the data directly without deserialising it first.
If sufficiently large amount of data is logged, it takes a while to see the last
event, because everything is deserialised in order. Additionally, something like
images cannot just be viewed with any other tool, since they are not saved as
image files. TensorBoard also limits the number of steps that can be viewed for
something for images or text, presumably to not blow up the serialised data, and
you cannot see anything in between or even decide which ones you want to keep.Lavdleverages the file system and the data can be freely viewed and edited
just like any other file, while still being picked up for the visualisation.
That also comes with its own drawbacks compared to serialising the data,
for example using more disk space, although that is almost certainly
a non-issue.Another difference to TensorBoard is that it does not revolve around TensorFlow
graphs or tensors or Machine Learning in general, but focuses more on the actual
data. Even though TensorBoard has been used for a while with PyTorch, the
graphs are not really the focus of that, but it's mostly used to log the
scalars, text, images etc. While the graph visualisation are nice, they are not
used all that often, because once the graph has been seen there is just no
reason to create it again, but there are countless experiments that use the
exact same graph. This also means thatLavddoes not try to cover everything
that TensorBoard does. Of course there are also features that TensorBoard does
not cover.Known IssuesInotify watch limit reachedOSError: [Errno 28] inotify watch limit reachedThe error occurs if there are too many files that are being watch at the same
time. You can check how many watches are allowed as follows:cat/proc/sys/fs/inotify/max_user_watches# => 8192The default is 8192 on most systems. To solve the issue the limit needs to be
increased:sudosysctlfs.inotify.max_user_watches=524288