Using PyTorchVideo for efficient video understanding | by Eric Hofesmann | Towards Data Science

The massive sea of computer vision models can be difficult to navigate if you are trying to find the best models or even just relevant baselines for your task. Model zoos like TensorFlow Hub and Facebook’s Detectron2 make it easy to access popular models. Further, libraries like PyTorch lightning make it easy to then modify these models to suit your needs. This is all well and good for images, but for videos, it’s another story. Video data is becoming increasingly more popular, but the additional complexity that comes with it often leaves video-related tasks on the backburner.
PyTorchVideo is a new library that has set out to make video models just as easy to load, build, and train as image models.
PyTorchVideo provides access to a video model zoo, video data processing functions, and a video-focused accelerator to deploy models all backed in PyTorch allowing for seamless integration into existing workflows.
The only thing missing from PyTorchVideo to complete your video workflows is a way to visualize your datasets and interpret your model results. This is where FiftyOne comes in. FiftyOne is an open-source tool that I have been working on at Voxel51. It is designed to make it easy to visualize any image or video dataset and explore ground truth and predicted labels stored locally or in the cloud. The flexible representation of FiftyOne datasets and the FiftyOne App let you quickly get hands-on with your datasets and interpret your models to find failure modes, annotation mistakes, visualize complex labels, and more.
This blog post is an extension of a recent PyTorchVideo tutorial and is written to teach you how to integrate PyTorchVideo with FiftyOne to close the loop on video-based ML workflows. Specifically, this post covers:
Downloading a subset of the Kinetics dataset
Loading a video dataset with FiftyOne
Using PyTorchVideo to perform inference
Visualizing and evaluating the PyTorchVideo model
You can run the examples in this blog post directly in your browser in this Google Colab notebook!
To follow along with this walkthrough, you will need to install FiftyOne, PyTorchVideo, PyTorch, and TorchVision:
pip install fiftyone pytorch torchvision
While PyTorchVideo is also installable through pip, the functionality in this post requires it to be installed through GitHub:
git clone https://github.com/facebookresearch/pytorchvideo.gitcd pytorchvideopip install -e .
This walkthrough uses a subset of the Kinetics-400 dataset which can be downloaded with the following code snippet:
One of the many reasons that video datasets are more difficult to work with than image datasets is the fact that many popular video datasets are available only through YouTube. So instead of being able to download a zip containing everything you need, you instead need to run scripts like the one below to download individual videos from YouTube that may or may not have become unavailable since the dataset was curated.
For image datasets, there are some rudimentary options available for visualizing batches of data like pillow and OpenCV. There are very few options available for visualizing video datasets. FiftyOne is a new open-source library that provides simple and powerful visualization for both image and video datasets.
If your dataset follows a common format, like the COCO format for detections, then you can load it in a single line of code:
Even if your dataset is in a custom format, it is still straightforward to load your dataset with FiftyOne. For example, if you are using an object detection video model, you can load your data as follows:
In this example, we will be following the PyTorchVision tutorial on running a video classification model. Generally, video classification datasets will be stored on disk in a directory tree whose subfolders define dataset classes. This format can be loaded in one line of code:
If you are following along yourself, hover over or click on the samples to play the videos:
We also need to download and store a list of default class names that will be used when evaluating predictions:
wget https://dl.fbaipublicfiles.com/pyslowfast/dataset/class_names/kinetics_classnames.json
In this section, we use PyTorchVideo to download and run a video classification model on the data that we loaded in the previous section and store the results in our dataset. The code in this section is adapted from this PyTorchVideo tutorial.
Torch Hub is a repository for pretrained PyTorch models that allow you to download models and run inference on your dataset. PyTorchVideo provides a number of video classification models through their Torch Hub-backed model zoo including SlowFast, I3D, C2D, R(2+1)D, and X3D. The following code snippet downloads the slow branch of SlowFast with a ResNet50 backbone and loads it into Python:
Every model has a specific input structure that it expects. The standard workflow is to write custom scripts that perform the necessary loading and transformation functions to format data for every model. PyTorchVideo expedites this process by providing these functions for you in a flexible way that will work for most video processing needs. For example, the following code constructs the transforms to sample frames from the video, normalize, scale, and crop it, without needing to write any of those functions yourself:
Since the dataset is being stored in FiftyOne, we can easily iterate through the samples, load and run our model on them with PyTorchVideo, and store the predictions back in FiftyOne for further visualization and analysis:
Aside from being an open-source ecosystem for dataset curation, FiftyOne is also designed to visualize, evaluate, and interpret models by allowing you to quickly find and address model failure modes.
To this end, we can start by visualizing the predictions generated in the last section:
We can then use FiftyOne to evaluate the predictions with the ground truth to view aggregate metrics and plots showing things like confusion matrices and precision-recall curves. This evaluation adds per-sample correctness labels (“eval”) to the dataset, which make it easy to filter by correct/incorrect predictions, or more generally by TP/FP/FN for object detections. Evaluation can be performed in just a single line of code:
                           precision    recall  f1-score   support       springboard diving       0.80      0.80      0.80         5            surfing water       1.00      0.60      0.75         5      swimming backstroke       1.00      0.80      0.89         5   swimming breast stroke       0.57      0.80      0.67         5swimming butterfly stroke       1.00      0.60      0.75         5                micro avg       0.82      0.72      0.77        25                macro avg       0.87      0.72      0.77        25             weighted avg       0.87      0.72      0.77        25
Let’s plot the confusion matrix for the classes we are interested in:
We can attach this plot to a session object to make it interactive. So if you click one of the cells, the FiftyOne App session updates to show the samples in that cell.
Note: Plots are currently only interactive in Jupyter Notebooks but additional environments will be supported soon!
FiftyOne also provides a novel query language to create views into your dataset by searching and filtering any given labels and metadata. This makes it easy to explore your dataset and find samples related to any question you may have in mind. For example, we can quickly find samples where the model was least certain about its prediction based on similar confidences across multiple classes and use the per-sample correctness labels (“eval”) from the previous evaluation to only look at incorrectly predicted samples:
Visualizing these samples lets us get an idea of the type of data that should be added to the training dataset. To mark these for future reference, we can use the tagging functionality in the FiftyOne App:
The ease of this hands-on analysis will generally lead to significant improvements in dataset quality, and consequently improvements in model performance, faster than any analysis only using aggregate dataset statistics.
While most large video datasets and research efforts revolve around classification problems like human activity recognition, applications of video-based ML often involve object detection. At the moment, PyTorchVideo primarily supports video classification problems, however, there are video object detection capabilities available in FiftyOne.
FiftyOne allows you to either generate predictions from an image-based object detection model in the FiftyOne Model Zoo or add predictions from your own model to a video dataset. There are a host of models available in the zoo. For example, let’s use EfficientDet-D0. We first need to install TensorFlow and AutoML. We can use the eta package that comes with FiftyOne to easily install AutoML:
pip install tensorflow==1.14eta install automl
Now let’s apply the model to a video and visualize the results:
This kind of visualization would require writing custom scripts to load the raw video, annotations, and predictions, then using software like OpenCV to draw boxes and export the visualizations to a new video on disk. Then if you want to change the labels you are looking at you would need to rewrite your script and regenerate the videos every time. Instead, all of this took us only a few lines of code and resulted in an easier-to-use and more flexible representation of our data.
Video-based machine learning models are growing in popularity but have lacked the same level of ease-of-use code bases that allow for quick development and evaluation of image models. PyTorchVideo aims to make it easier to implement, train, and evaluate video models through their model zoo, video-focused components, and acceleration functions. On the flip side, where PyTorchVideo is making it easier to work with video models, FiftyOne is an open-source library that aims to make it easy and efficient to curate, evaluate, and improve video (and image) datasets. Together, FiftyOne and PyTorchVideo provide significant savings in the time and effort required to create high-quality video datasets and models.
Disclosure: I work at Voxel51 and am a developer of FiftyOne
Headquartered in Ann Arbor, Michigan, and founded in 2016 by University of Michigan professor Dr. Jason Corso and Dr. Brian Moore, Voxel51 is an AI software company that is democratizing access to software 2.0 by providing the open core software building blocks that enable computer vision and machine learning engineers to rapidly engineer data-powered workflows.
Learn more at fiftyone.ai!