Fixing HTML Video on Mobile
By: Samir Zahran
Tags: JavaScript, Video, Python, Mobile, Web Development

The problem with mobile video

In early 2015 we started development on a redesign for This Also’s website. As the project evolved it became clear that video would be a major component of the design language. We had a large background video on the home page and footer, more large videos near the top of each project page and many smaller videos littered across most of the other pages. All of them were meant to either autoplay or play and pause themselves due to some scripted behavior. None of them required any user interface or direct interaction.

The thisalso.com homepage

These requirements were already complex on desktop, but on mobile they were a nightmare. Common HTML5 Video features such as preloading and autoplay are completely missing in some browsers. The scripting APIs are limited compared to what’s available on desktop. Worst of all, Safari on the iPhone (the most popular mobile browser to visit our site) does not allow inline video playback at all (Note: This limitation is being lifted in iOS 10 which is set to be released later this Fall).

GIFs weren’t an ideal workaround and requiring users to open a video in fullscreen didn’t make sense for the kinds of videos we were using. We looked for a solution and discovered other people had already devised some clever solutions.

Apple was doing it. So were a handful of news sites and other agencies. But none of them (that we could find) published their solutions or made them available to the community as a whole.

So we made our own

Today we are releasing Whitewater Mobile Video. It consists of an encoder written in Python and a player written in Javascript. Both are free to use and open source.

Instructions for how to use them are detailed elsewhere, so what we’ll do here is explain how they work so that you can decide whether Whitewater is appropriate for your own projects.

Faking Video 101

The basics of simulating a video in the browser are fairly simple. If you draw a sequence of images inside of a <canvas> tag fast enough, that’s a video. But that means creating those image assets up-front. You could save each frame as a separate image, but that’s neither practical nor efficient for lengthy or hi-resolution videos. To come up with a more efficient method for making those images there were three main considerations that we balanced:

Overall file size of all assets combined, which is an important factor for mobile devices with limited RAM The number of HTTP requests needed to load all of the required assets, which can have a huge impact on the performance of a page as it loads The amount of computational complexity on the browser required to process and recreate each video frame

We prioritized reducing file size and number of downloads at the cost of a more complex playback process

Whitewater favors optimizing 1 & 2 over 3. The complexity of the player might cause some slowdown on weaker devices, but loading oversized and/or too many assets is a more likely cause of browser crashes. As a result, the encoding — and therefore, reassembly—was necessarily more complex.

So how does encoding work?

The job of the encoder is twofold: 1: It saves the visual assets needed to reassemble the video, and 2: it provides a set of instructions for doing so. Because we want to minimize file sizes and asset count, our goal is to to cut out as much duplicate visual information as possible and then condense what’s left. The encoder accomplishes this by comparing each frame to the frame preceding it and saving only the parts that have changed.