Package: vp-suite
IntroductionVideo prediction ('VP') is the task of predicting future frames given some context frames.Like with most Computer Vision sub-domains, scientific contributions in this field exhibit a high variance in the following aspects:Training protocol(dataset usage, when to backprop, value ranges etc.)Technical details of model implementation(deep learning framework, package dependencies etc.)Benchmark selection and execution(this includes the choice of dataset, number of context/predicted frames, skipping frames in the observed sequences etc.)Evaluation protocol(metrics chosen, variations in implementation/reduction modes, different ways of creating visualizations etc.)Furthermore, while many contributors nowadays do share their code, seemingly minor missing parts such as dataloaders etc. make it much harder to assess, compare and improve existing models.This repo aims at providing a suite that facilitates scientific work in the subfield, providing standardized yet customizable solutions for the aspects mentioned above. This way, validating existing VP models and creating new ones hopefully becomes much less tedious.InstallationRequirespipandpython >= 3.6(code is tested with version3.8).From PyPipip install vp-suiteFrom sourcepip install git+https://github.com/Flunzmas/vp-suite.gitIf you want to contributegit clone https://github.com/Flunzmas/vp-suite.git
cd vp-suite
pip install -e .[dev]If you want to build docsgit clone https://github.com/Flunzmas/vp-suite.git
cd vp-suite
pip install -e .[doc]UsageChanging save locationWhen using this package for the first time, the save location for datasets,
models and logs is set to<installation_dir>/vp-suite-data.
If you'd like to change that, simply run:python vp_suite/resource/set_run_path.pyThis script changes your save location and migrates any existing data.Training modelsfromvp_suiteimportVPSuite# 1. Set up the VP Suite.suite=VPSuite()# 2. Load one of the provided datasets.#    They will be downloaded automatically if no downloaded data is found.suite.load_dataset("MM")# load moving MNIST dataset from default location# 3. Create a video prediction model.suite.create_model('convlstm-shi')# create a ConvLSTM-Based Prediction Model.# 4. Run the training loop, optionally providing custom configuration.suite.train(lr=2e-4,epochs=100)This code snippet will train the model, log training progress to yourWeights & Biasesaccount,
save model checkpoints on improvement and generate and save prediction visualizations.Evaluating modelsfromvp_suiteimportVPSuite# 1. Set up the VP Suite.suite=VPSuite()# 2. Load one of the provided datasets in test mode.#    They will be downloaded automatically if no downloaded data is found.suite.load_dataset("MM",split="test")# load moving MNIST dataset from default location# 3. Get the filepaths to the models you'd like to test and load the modelsmodel_dirs=["out/model_foo/","out/model_bar/"]formodel_dirinmodel_dirs:suite.load_model(model_dir,ckpt_name="best_model.pth")# 4. Test the loaded models on the loaded test sets.suite.test(context_frames=5,pred_frames=10)This code will evaluate the loaded models on the loaded dataset (its test portion, if avaliable),
creating detailed summaries of prediction performance across a customizable set of metrics.
The results as well as prediction visualizations are saved and logged toWeights & Biases.Note 1: If the specified evaluation protocol or the loaded dataset is incompatible with one of the models,
this will raise an error with an explanation.Note 2: By default, aCopyLastFramebaseline is also loaded and tested with the other models.Hyperparameter OptimizationThis package usesoptunato provide hyperparameter optimization functionalities.
The following snippet provides a full example:importjsonfromvp_suiteimportVPSuitefromvp_suite.defaultsimportSETTINGSsuite=VPSuite()suite.load_dataset(dataset="KTH")# select dataset of choicesuite.create_model(model_id="lstm")# select model of choicewithopen(str((SETTINGS.PKG_RESOURCES/"optuna_example_config.json").resolve()),'r')ascfg_file:optuna_cfg=json.load(cfg_file)# optuna_cfg specifies the parameters' search intervals and scales; modify as you wish.suite.hyperopt(optuna_cfg,n_trials=30,epochs=10)This code e.g. will run 30 training loops (calledtrialsby optuna), producing a trained model for each hyperparameter configuration and writing the hyperparameter configuration of the best performing run to the console.Note 1: For hyperopt, visualization, logging and model checkpointing is minimized to reduce IO strain.Note 2: Despite optuna's trial pruning capabilities, running a high number of trials might still take a lot of time.
In that case, consider e.g. reducing the number of training epochs.Useno_wandb=True/no_vis=Trueif you want to log outputs to the console instead/not generate and save visualizations.Notes:UseVPSuite.list_available_models()andVPSuite.list_available_datasets()to get an overview of which models and datasets are currently covered by the framework.All training, testing and hyperparametrization calls can be heavily configured (adjusting training hyperparameters, logging behavior etc, ...).
For a comprehensive list of all adjustable run configuration parameters see the documentation of thevp_suite.defaultspackage.CustomizationThis package is designed with quick extensibility in mind. See the sections below for how to add new components
(models, model blocks, datasets or measures).New ModelsCreate a file<your name>.pyin the foldervp_suite/models.Create a class that derives fromvp_suite.base.base_model.VideoPredictionModeland override/specify new constants you need.Write your model code or import existing code so that the superclass interface is still served.
If desired, you can implement a custom training/evaluation loop iterationtrain_iter()/eval_iter()that gets called instead of the default training/evaluation loop iteration.Register your model in theMODEL_CLASSESdictionary ofvp_suite/models/__init__.py, giving it a key that can be used by the suite.
By now, you should be able to create an instance of your model withVPSuite.create_model()and train it on a dataset withVPSuite.train().New Model BlocksCreate a file<your name>.pyin the foldervp_suite/model_blocks.Create a class that derives fromvp_suite.base.base_model_block.ModelBlockand override/specify new constants you need.Write your model block code or import existing code so that the superclass interface is still served.If desired, add a local import of your model block tovp_suite/model_blocks/__init__.py(this registers the model block package-wide).New DatasetsCreate a file<your name>.pyin the foldervp_suite/datasets.Create a class that derives fromvp_suite.base.base_dataset.BaseVPDatasetand override/specify new constants you need.Write your dataset code or import existing code so that the superclass interface is served.
If it's a public dataset, consider implementing methods to automatically download it.Register your dataset in theDATASET_CLASSESdict ofvp_suite/dataset/__init__.py, giving it a key that can be used by the suite.
By now, you should be able to load your dataset withVPSuite.load_dataset()and train models on it withVPSuite.train().New measures (losses and/or metrics)Create a new file<your name>.pyin the foldervp_suite/measure, containing your loss or metric.Makevp_suite.base.base_measure.BaseMeasureits superclass and override/implement all needed implementations and constants.Register the measure in theMETRIC_CLASSESdict ofvp_suite/measure/__init__.pyand, if it can also be used as a loss, in theLOSS_CLASSESdict.Notes:If you omit the docstring for a particular attribute/method/field, the docstring of the base class is used for documentation.If implementing components that originate from publications/public repositories, please override the corresponding constants to specify the source!
Additionally, if you want to write automated tests checking implementation equality,
have a look at howtests/test_impl_match.pyfetches the tests oftests/test_impl_match/and executes these tests.Basic unit tests for models, datasets and measures are executed on all registered models -
you don't need to write such basic tests for your custom components!
Same applies for documentation: The tables that list available components are filled automatically.ContributingThis project is always open to extension! It grows especially powerful with more models and datasets, so if you've made your code work on custom models/datasets/metrics/etc., feel free to submit a merge request!Other kinds of contributions are also very welcome - just check the open issues on thetrackeror open up a new issue there.Unit TestingWhen submitting a merge request, please make sure all tests run through (execute from root folder):python -m pytest --runslow --cov=vp_suiteNote: this is the easiest way to run all testswithout import hassles.
You will need to havevp-suiteinstalled in development move, though (see here).API DocumentationThe official API documentation is updated automatically upon push to the main branch.
If you want to build the documentation locally, make sure you've installed the packageaccordinglyand execute the following:cd docs/
bash assemble_docs.shAcknowledgementsProject structure is inspired bysegmentation_models.pytorch.Sphinx-autodoc templates are inspired by theQNETrepository.All other sources are acknowledged in the documentation of the respective point of usage (to the best of our knowledge).LicenseThis project comes with anMIT License, except for the following components:Modulevp_suite.measure.fvd.pytorch_i3d(Apache 2.0 License, taken and modified fromhere)DisclaimerI do not host or distribute any dataset. For all provided dataset functionality, I trust you have the permission to download and use the respective data.