Package: pyflor
FLOR: Experiment Management for ML EngineersYou can use FLOR to take checkpoints during model training.
These checkpoints allow you to restore arbitrary training data post-hoc and efficiently,
thanks to memoization and parallelism speedups on replay.FLOR is a suite of machine learning tools for hindsight logging.
Hindsight logging is an optimistic logging practice favored by agile model developers.
Model developers log training metrics such as the loss and accuracy by default,
and selectively restore additional training data --- like tensor histograms, images, and overlays --- post-hoc,
if and when there is evidence of a problem.FLOR is software developed at UC Berkeley'sRISELab,
and is being released as part of an accompanyingVLDB publication.InstallationpipinstallpyflorFLOR expects a recent version of Python (3.7+) and PyTorch (1.0+).gitcheckout-bflor.shadow
python3examples/linear.py--florlinearRun thelinear.pyscript to test your installation.
This script will train a small linear model on MNIST.
Think of it as a ''hello world'' of deep learning.
We will cover FLOR shadow branches later.ls~/.flor/linearConfirm that FLOR saved checkpoints of thelinear.pyexecution on your home directory.
FLOR will access and interpret contents of~/.florautomatically.
Do watch out for storage footprint though.
If you see disk space running out, check~/.flor.
FLOR includes utilities for spooling its checkpoints toS3.Preparing your Training ScriptfromflorimportMTKasFlorforepochinFlor.loop(range(...)):...First, wrap the iterator of the main loop with FLOR's generator:Flor.loop.
The generator enables FLOR to parallelize replay of the main loop,
and to jump to an arbitrary epoch for data recovery.fromflorimportMTKasFlorimporttorchtrainloader:torch.utils.data.DataLoadertestloader:torch.utils.data.DataLoaderoptimizer:torch.optim.Optimizernet:torch.nn.Modulecriterion:torch.nn._LossFlor.checkpoints(net,optimizer)forepochinFlor.loop(range(...)):fordatainFlor.loop(trainloader):inputs,labels=dataoptimizer.zero_grad()outputs=net(inputs)loss=criterion(outputs,labels)loss.backward()optimizer.step()print(f"loss:{loss.item()}")eval(net,testloader)That's it, your training code is now ready for record-replay.Training your modelpython3training_script.py--florNAME[your_script_flags]Before we train your model,make sure that your model training code is part of a git repository.
Model training is exploratory and it's common to iterate dozens of times
before finding the right fit.
We'd hate for you to be manually responsible for managing all those versions.
Instead, we ask you to create a FLOR shadow branch
that we can automatically commit changes to.
Think of it as a sandbox: you get the benefits of autosaving,
without worrying about us poluting your main branch with frequent & automatic commits.
Later, you can merge the changes you like.In FLOR, all experiments need a name.
As your training scripts and configurations evolve,
keep the same experiment name so FLOR
associates the checkpoints as versions of the same experiment.
If you want to re-use the name from the previous run,
you may leave the field blank.Hindsight LoggingfromflorimportMTKasFlorimporttorchtrainloader:torch.utils.data.DataLoadertestloader:torch.utils.data.DataLoaderoptimizer:torch.optim.Optimizernet:torch.nn.Modulecriterion:torch.nn._LossforepochinFlor.loop(range(...)):forbatchinFlor.loop(trainloader):...eval(net,testloader)log_confusion_matrix(net,testloader)Suppose you want to view a confusion matrix as it changes
throughout training.
Add the code to generate the confusion matrix, as sugared above.python3training_script.py--replay_florYou first switch to the FLOR shadow branch,
and select the version you wish to replay
from thegit loglist.
In our example, we won't checkout version,
because we want to replay the latest version,
which is selected by default.You will tell FLOR to replay by setting the flag--replay_flor.
FLOR is performing fast replay, so you may generalize this
example to recover ad-hoc training data.
In our example, FLOR will compute your confusion matrix
and automatically skip the nested training loop
by loading its checkpoints.fromflorimportMTKasFlorimporttorchtrainloader:torch.utils.data.DataLoadertestloader:torch.utils.data.DataLoaderoptimizer:torch.optim.Optimizernet:torch.nn.Modulecriterion:torch.nn._LossforepochinFlor.loop(range(...)):forbatchinFlor.loop(trainloader,probed=True):...eval(net,testloader)log_confusion_matrix(net,testloader)Now, suppose you also wantTensorBoardto plot the tensor histograms.
In this case, it is not possible to skip the nested training loop
because we are probing intermediate data.
We tell FLOR to step into the nested training loop by settingprobed=True.Although we can't skip the nested training loop, we can parallelize replay or
re-execute just a fraction of the epochs (e.g. near the epoch where we see a loss anomaly).python3training_script.py--replay_florPID/NGPUS[your_flags]As before, you tell FLOR to run in replay mode by setting--replay_flor.
You'll also tell FLOR how many GPUs from the pool to use for parallelism,
and you'll dispatch this script simultaneously, varying thepid:<int>to span all the GPUs. To run segment 3 out of 5 segments, you would write:--replay_flor 3/5.If instead of replaying all of training you wish to re-execute only a fraction of the epochs
you can do this by setting the value ofngpusandpidrespectively.
Suppose you want to run the tenth epoch of a training job that ran for 200 epochs. You would setpid:9andngpus:200.We provide additional examples in theexamplesdirectory. A good starting point islinear.py.PublicationsTo cite this work, please refer to theHindsight Loggingpaper (VLDB '21).FLOR is open source software developed at UC Berkeley.Joe Hellerstein(databases),Joey Gonzalez(machine learning), andKoushik Sen(programming languages)
are the primary faculty members leading this work.This work is released as part ofRolando Garcia's doctoral dissertation at UC Berkeley,
and has been the subject of study by Eric Liu and Anusha Dandamudi,
both of whom completed their master's theses on FLOR.
Our list of publications are reproduced below.
Finally, we thankVikram Sreekanti,Dan Crankshaw, andNeeraja Yadwadkarfor guidance, comments, and advice.Bobby Yanwas instrumental in the development of FLOR and its corresponding experimental evaluation.Hindsight Logging for Model Training.R Garcia, E Liu, V Sreekanti, B Yan, A Dandamudi, JE Gonzalez, JM Hellerstein, K Sen. The VLDB Journal, 2021.Fast Low-Overhead Logging Extending Time.A Dandamudi. EECS Department, UC Berkeley Technical Report, 2021.Low Overhead Materialization with FLOR.E Liu. EECS Department, UC Berkeley Technical Report, 2020.LicenseFLOR is licensed under theApache v2 License.