Unsupervised Structure-from-motion (SfM) Demystified
By: Yasin Almalioglu
Tags: Artificial Intelligence, Deep Learning, Autonomous Vehicles, Robotics, Unsupervised Learning

Photo by Franki Chamaki on Unsplash

Given that you found this page, I assume you somehow heard about structure from motion (SfM), or you are an expert on it. You also heard about deep learning but wonder that “What on earth is this unsupervised SfM?”. Well, carry on reading, then. You are in the right spot.

A Wikipedia-like pseudo-definition of unsupervised SfM would be something like:

Unsupervised SfM is an ego-motion and 3D structure estimation method from 2D image sequences based on deep learning without any need for supervision.

Let’s first give some background on deep learning-based SfM. In a typical deep learning method, you have a magical black box. It takes your image sequences and gives you the output you expected, for example, the camera motion or 3D structure. Of course, that does not happen in no time. You need to teach the tricks to the black box. One way to do that is to use your existing expensive, bulky or non-portable systems that you cannot use everywhere. For example, you might have an expensive, highly accurate motion tracking system or 3D laser scanner that you can use at your home or lab. Still, you cannot put in your pocket and take it anywhere you go. Another example, you already have a dataset that already contains this expensive information. Still, you want to predict this information on your dataset. In either case, this is your trick! You use costly information to teach you black box. Let’s crack this supervised black box open in another story. Let’s now continue with the unsupervised one, which is why you are here.

Unsupervised SfM: Idea

What if your work is much more challenging and you don’t have a chance to accurately track your camera or scan your environment for many good reasons. For example, your system might be supposed to work in the wild like a self-driving car. Maybe, it should work in a dangerous environment like an underground tunnel. Don’t give up! Here comes the unsupervised SfM for you. Actually, for a long time, we know how cameras work and how camera motion changes the image pixels. Why don’t we use this information as the trick? We don’t even need stereo cameras. This idea, view reconstruction, has attracted significant attention in recent years to learn the 3D structure and ego-motion estimators from monocular image sequences.

The idea with the view reconstruction is that if we know the depth of images in your video and camera motion during the video recording, we can reconstruct our colour images. So, camera motion and 3D structure are geometrically tied to our images through our camera parameters (camera intrinsics). You can think of this as a function that calculates a colour image from the given depth, camera motion and neighbour images. If you are curious further about it and want to know about advanced stuff, I recommend a few papers at the bottom.

The view reconstruction constraint provides a supervision signal. It forces the neural network to synthesize a target image from multiple source images acquired from different angles.

Unsupervised SfM in Action

All you have is just a video, and you can convert it to 3D and see how your camera moves. Sounds fabulous! Let’s look at some of the existing cool work.

DPVO [4] takes the unsupervised SfM to the next level: Unsupervised SfM in extreme environments like an old nuclear power plant in the DARPA Subterranean Challenge.

Here is a recent SC-SfMLearner [1] example of the unsupervised SfM:

Suppose you also do not know about your camera parameters. In that case, these guys offer a solution for that Depth from Video in the wild [2].

If you are interested in Generative Adversarial Models and Recurrent Neural Networks and how to use them in unsupervised SfM, here is GANVO [3].

I hope you have just survived and enjoyed all the reading upto this point. Although SfM is well-established topic that is of interest to many communities, learning SfM in an end-to-end manner in unsupervised way is far more goergous and challenging than conventional methods. So, happy coding!

Recommendations