High-quality slow-motion videos in 5 minutes with Deep Learning | by Dimitris Poulopoulos | Towards Data Science

We can all agree that slow-motion videos create this dramatic effect, which certainly adds an extra flavor to specific scenes. On the other hand, slowing down a video might create several unwanted artifacts, which could deem it unwatchable. But what makes a slow-motion video great?
Let’s use Deep Learning for free, to produce the smooth result of a high-end camera capable of recording at high frame-rates
Among other things, I would argue that the most important aspect is its FPS count: how many frames per second are rendered to the screen. Too low and the video stutters, creating an unpleasant viewing experience. Thus, we need to have an expensive camera able to record at a high frame-rate. Or, we could use Deep Learning for free, and try to synthesize intermediate images between two consecutive frames.
Super slow-motion is just one of the applications of Video Frame Interpolation or VFI as it’s commonly known in the field of Computer Vision. In this story, we present RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation. We touch on what it is bringing to the table and take it for a quick test drive using our own videos. Let’s dive in!
Learning Rate is a newsletter for those who are curious about the world of AI and MLOps. You’ll hear from me every Friday with updates and thoughts on the latest AI news and articles. Subscribe here!
The ultimate goal for Video Frame Interpolation (VFI) is to improve the frame rate and enhance the visual quality of a video. To achieve this, it tries to synthesize intermediate images between two consecutive frames, that are both spatially and temporally coherent with the existing context.
In the core of VFI, we find another vision task named Optical Flow. So, let’s take a quick detour to see what Optical Flow means, and how it helps us with VFI. Then, we will be ready to watch RIFE in action and play with it using our own videos.
Very broadly, Optical Flow is understanding how things are moving in an image, at the pixel level. This is useful for a number of downstream video editing tasks, such as video analysis, image stabilization, and video alignment.
An image is comprised of many pixels, and when we move our camera around there is some kind of change happening in these pixels. Optical Flow tries to figure out how that motion affects each individual pixel. So, we are trying to attach a motion vector to each pixel in the image, indicating where this pixel is moving to.
Having that information is key to understand what is happening in an image, how objects are moving, and in what direction. Then, the next step is to generate plausible frames in-between these pixel changes. This is where VFI comes into play.
We saw what VFI is and how it can increase the visual quality of a video. We also saw what Optical Flow is, why it is considered a core element of VFI, and how it can help us solve it. Now, it’s time to introduce RIFE and what it brings to the table.
RIFE uses a neural network named IFNet that can directly estimate the intermediate flows from images. The main contributions of this new work are summarized below:
The design of a novel and efficient network architecture, to simplify the flow-based VFI methods. Thus, given two input frames, the proposed model can be trained from scratch and directly approximate the intermediate flows.
A novel leakage distillation loss function, which leads to a more stable convergence and large performance improvement
The first flow-based real-time VFI algorithm that can process 720p videos at 30FPS
Below we can see some examples of videos processed with RIFE. On the left, we see the original video, and on the right its enhanced version. Watch how smoother the processed version is.
For more details, you can read the paper on ArXiv and browse through the project’s page. Let’s get into the fun part: use RIFE on our own videos!
Using RIFE on our own videos it’s really easy. However, if we really want it to shine, we need an Nvidia GPU. To this end, let’s use Google Colab!
Got to colab.research.google.com and create a new Notebook, with GPU support. If you haven’t done this before, select Change Runtime Type from the Runtime menu and select GPU.
Now, we are ready to clone the RIFE repo. In a cell type the following:
!git clone https://github.com/hzwer/arXiv2020-RIFE
Next, we should download the pre-trained models, so we can generate predictions for the video’s frames. To achieve this, we use a Python’s module that makes it easy to download objects from Google Drive: gdown
!gdown --id 1zYc3PEN4t6GOUoVYJjvcXoMmM3kFDNGS!7z e RIFE_trained_model_new.zip
We have now downloaded the model’s weights and extracted them into our workspace. Let’s move them to the folder that the software expects them to be.
!mkdir /content/arXiv2020-RIFE/train_log!mv *.pkl /content/arXiv2020-RIFE/train_log/
Next, we should install the project’s dependencies. cd into the project’s folder and use pip to install the necessary packages:
%cd /content/arXiv2020-RIFE/!pip3 install -r requirements.txt
Finally, upload your video into the project’s folder and run the following command:
!python3 inference_video.py --exp=2 --video=mydemo.mp4
This script will quadruple the number of frames in your video. The output will be stored in the same location and its name will be something like mydemo_4X_120fps.mp4. Congratulations! Your new, enhanced video is now ready!
There are some other options you can pass to the script. One of the most useful ones is --montage, which creates a montage between the original and the generated videos. Try it to compare the two items easily.
In this story, we saw what Optical Flow is and how it is related to Video Frame Interpolation. We examined what are the goals of these two methods, and how a new technique, RIFE: Real-Time Intermediate Flow Estimation for VFI, can be used to generate impressive new content.
Finally, we played with RIFE and our own videos. I kept the best part for the end: there is already a new Windows app that uses RIFE to provide Fast Video Interpolation for any GPU: Flowframes. However, using python you can reap the benefits of RIFE on any GPU-enabled machine!
My name is Dimitris Poulopoulos, and I’m a machine learning engineer working for Arrikto. I have designed and implemented AI and software solutions for major clients such as the European Commission, Eurostat, IMF, the European Central Bank, OECD, and IKEA.
If you are interested in reading more posts about Machine Learning, Deep Learning, Data Science, and DataOps, follow me on Medium, LinkedIn, or @james2pl on Twitter.
Opinions expressed are solely my own and do not express the views or opinions of my employer.