The VR Industry, Part 1: Cinematic VR
By: Armando Kirwin
Tags: Virtual Reality, Storytelling, VR, Augmented Reality, 360 Video

And that wasn’t even the hardest part. In addition to hardware engineering limitations, there was another major hurdle: software. VR cameras are made up of multiple small cameras arranged in a sphere. After you’re done shooting, you have to blend all of those cameras together into something seamless. This was known as “stitching” and it was done manually by VFX artists.

Stitching is an extremely labor intensive process. Imagine trying to blend 8, 17, or even 24 different cameras so that they all have the same exposure, color balance, etc. for every single frame, 30 or 60 frames per second, in 3D. In fact, because this was mostly manual labor, it cost between $8000-$20,000 per minute of final product (e.g. a five minute film would cost $40,000-$100,000 just to stitch). It also took about one week per minute to complete.

Solving the stitching problem required the development of complex software algorithms and once again no one, including startups with tens of millions of dollars in funding, had the expertise they needed to do right. Unfortunately, some of them pretended that they could and it ended up hurting the industry because it downplayed the true cost and difficulty of making cinematic VR — a trend that continues to this day. The truth is that only large companies like Facebook and Google were in a position to try, and the good news is that they stepped in and did it after about two years.

However, not even these mighty giants were able to fully solve algorithmic stitching (which is achieved using something called optical flow). I would subjectively say that optical flow solutions are currently reaching a success rate of around 80–90%, with Google’s Jump assembler being the market leader. This makes sense because, generally speaking, algorithms are Google’s speciality.

A recent project that I did, which at first seems like a simple film noir series starring famous actors, is in fact a technical showcase of what you can achieve using nearly 100% optical flow with minimal manual VFX. And because we had almost 20 minutes of final product, we saved The New York Times about a quarter million dollars in stitching (we also finished all nine films in only three weeks). This is why it makes sense that VR professionals are willing to spend $15,000 on the GoPro Odyssey camera (which is compatible with the Jump assembler) or others like it.

VR camera improvement over time: (L) GoPro Odyssey modified with external top camera, third-party wireless preview, and external battery (not pictured). (R) Yi HALO with fully integrated top camera, preview, and battery.

The GoPro Odyssey was only step one. As one of the first and only people in the world to use the best optical flow cameras on the market (including the Facebook Surround 360 and Google’s upcoming Yi HALO), I predict that optical flow-based solutions are one or two generations away from being completely reliable for most forms of production. This means that in ~18 months anyone will have access to the same caliber of technology as the very best proprietary systems held by a handful of companies today.

Even more exciting is that I am only aware of two proprietary systems in the world that are objectively better than the current generation of optical flow cameras and the quality gap isn’t even very large.