Futatabi: Multi-camera instant replay
with slow motion
Steinar H. Gunderson
FOSDEM, February 2nd 2019

Hi! Welcome to the speaker notes for mypresentation about Futatabi, my instant replay andslow motion system. As you might notice fromseeing the slides, they don’t always make a lot ofsense without the notes—they don’t necessarily
match exactly what I’m going to say, but theyshould hopefully be of good use.
This talk is going to touch on two of my big passions,
namely programming and ultimate (also known as
ultimate frisbee). Ultimate isn’t a big sport in
Norway, so when I talk about it, I often have toexplain that it’s a team sport with a frisbee, wherethe goal is to pass to a teammate in the end zone.

Unfortunately, when I moved back home to Norway
and started playing tournaments, the only available
stream looked like this. Running on Xsplit inWindows, it was a very uneven 15 fps or so, andworse, everything was smudged together. There’s
supposed to be a goal line there (it’s at the sixmeter line), but you can’t see it due to the poor
image quality—it’s just very hard to explain a gameto people when you can’t even see whether a pass
was caught in or out.
Naturally, the question became: Could we do betterwith free software?

It turns out we can. This is Nageru, my live video
mixer. I presented it at FOSDEM 2016, and it’sgrown a lot since then. We’d never done sports
before, but it sounded like a fun challenge, so I
brought a few friends to see what we could do.

1902 mm

Goal

⚡

Goal

1188 mm1349 mm⚡2268 mm2666 mm2520 mmLet’s have a look at what the physical setup looks
like. This is a standard indoor ultimate field,
40x20m (ultimate is mostly played outdoors in mostparts of the world, but Norway primarily has anindoor tradition, using handball fields) plus a tiny bitof space on all sides.

1902 mm

Goal

⚡

Goal

1188 mm1349 mm⚡

2268 mm2666 mm

12520 mm

We don’t have a large team. The producer operates
Nageru with one hand and operates camera 1,
mostly panning, with the other. They also mix audio
and control the overlay graphics.
The stream is pointed to camera 1 maybe 80–85% ofthe time. It’s a general do-everything angle thatshows most of the active parts of the field as theplay progresses.
You can see the table for the commentators, whohave a comfort output of the stream.

1902 mm

Goal

⚡

Goal

1188 mm1349 mm⚡

2268 mm2666 mm

212520 mm

Camera 2 is run by a separate operator. Whilecamera 1 shows the entire play, camera 2 is usuallyzoomed in and following the player with disc. It canbe a challenging task for the operator, but it allows
for detail shots that show what camera 1 can’t.1902 mm

Goal

⚡

Goal

1188 mm341349 mm\\⚡

2268 mm2666 mm

212520 mm

Finally, we have two goal cameras (3 and 4) for“beauty shots”—we generally don’t use them a lot
during the actual point, but they can be used inbetween. They are GoPros, mounted high up
above each end zone on cheap, static tripod
extenders. Depending on the venue, we also
sometimes have an ambience cam for other fields,a camera on the commentators, or a smartphone
as a mobile IP camera.
As a trick, we run these and their associated SDI
converters on power banks—they last more than
12 hours, so we don’t have to run power to them.
Most equipment is either stuff we had already,borrowed or bought second-hand on eBay, so it’sreally a shoestring budget. And it’s nearly 100%
free software, all the way to the JavaScript on theplayer page.

This is a video of what it looks like. Note the HTML5graphics in the top left—the data is fed directly fromthe scoreboard over serial port. (If there’s twofields, we show both scores.) We also pull data livefrom Google Sheets for lower thirds, realtime
updated tables, and so on. And you can see thegray score line!
However, we soon noticed that it’s not enough.
People would make marginal catches or step reallynear the line, and even after all this work, we still
don’t know whether they were in or out. We neededsome form of instant replay, and it needed to be inslow motion.

So we went online to see what a slow motion
appliance would cost. This is the EVS XT3, which isthe standard choice in broadcast. Pretty much
every time you see a major sports event on TV,
there’s an XT3 in the chain. Or multiple ones.

Unfortunately, it’s £99,000. Used. (Plus £30 inshipping to Norway. YMMV.) And to add insult toinjury, the seller’s on vacation. Even the remotecontrol is thousands of Euros!
There are cheaper devices available, but you’regenerally in the €10k+ range. It’s clear that we can’t
go down this route—we need a software-basedsolution.

Enter Futatabi. It works as a multi-track recorder—
everything is recorded, all the time. It gets theframes from Nageru over a standard network; I’ve
chosen MJPEG as the codec, as it’s reasonably
high-quality and has hardware encoders/decoders
in recent Intel GPUs. We have roughly 75 Mbit/sec/
camera in 720p, or 125 GB/hour for four cameras.
You can then make clips, put the into playlists andplay them back, again over TCP/IP. A nice bonus isthat if we don’t want to modify the frames, we canecho them right back over the socket, byte for byte,
giving the best possible quality and speed.
(Futatabi works to maximize the usage of theseframes, but if you’re playing back at non-integral
rates, e.g. ramping, it’s not always possible.)
But when we slow down, how do we fill the gaps
between the frames? Let’s look at the alternatives.

We can try to just repeat each frame the required
amount of times, but the result isn’t so pretty—it’sstuttering visibly. (If your PDF viewer supports
video, you can click to play it. But not on the notes
page—you’ll need to go to the slides PDF. Sorry.)
Note that I’ve purposefully chosen a challengingsample here; there’s lots of motion, and I’m alsousing 0.25x (“super-slow”) instead of the regular
0.5x. There are two reasons: First of all, it’s a lot
more interesting to look at something that goes
wrong—who cares about the easy case? Second, if
you’re not using to look at these various forms ofartifacts, it’s good if they are clear. Also, theFOSDEM stream is downconverted to 25 fps, so it’sfor the stream viewers, too.

Fading between each frame isn’t much better. It’s justas choppy, with some extra blur to boot.

To illustrate, I’ve overlaid two consecutive frames
from the previous video. It’s obvious that creating
good in-between frames somehow has to relate tomotion.

This brings us to the basic idea of optical flow. For
every pixel in the first frame, we can try to estimate
where it moved in the second one. It’s not a perfectmodel—perhaps it went to nowhere (occlusion),
and with things like blur, maybe half of it went to
one place and half of it went to somewhere else.
But it’s a fairly good model, if we can make a good
estimate.
In this example, most of the vectors are slightly up
and to the right, since that’s how the camera
moved. But the disc is moving faster to the right,
and the player is moving to the left.

So once we have a flow field between the twoframes, we can halve and then somehow invert it.
(We won’t be discussing this step in detail—it’s a bit
trickier than it looks at first sight.) This would giveus, for the intermediate frame, where to fetch eachpixel from, which is more directly useful than where
it wants to go.
Note that this assumes near-linear motion, but forhigh enough frame rates, it’s a fairly good
approximation.

There are more than 200 different papers about
optical flow! (There are also lots of other ways tointerpolate frames; in particular, deep learning
methods are rapidly becoming popular.) I readthrough a bunch of the most promising one.
Unfortunately, realtime optical flow isn’t of all thatmuch interest in academia, but I found one thatshowed a lot of promise. It talks about 300–600 fps
on a single CPU core, and it comes with reference
code.
Now, 300–600 fps isn’t really what you get inpractice; on reasonable settings, it’s more like 10fps. I ended up making a GPU reimplementationfrom scratch, also for license reasons (thereference code was a bit unclear on licensing).

Let’s go through the algorithm on a high level. We
start with a motion search on low resolution; divideup the image in a series of overlapping blocks, andsearch for a similar block in the other image.
This motion search isn’t quite what you’d see in avideo codec—in particular, we’re interested in real
motion, not the best mathematical match. The DIS
paper uses gradient descent: We start with a zero
motion vector and then try to move in the directionof the right change. Assuming the motion is small
compared to the resolution (which it is!), we have areasonable chance of finding the right vector.
Note that we’re not drawing the motion vectors as
arrows anymore, since it’s hard to read lots of tinyarrows. Instead, we’re using the hue to denote theangle, and the lightness to show the length. In thiscase, the flow vector is a bit to the right.

After motion search over all the blocks, we have ourinitial field. Here the overlapping blocks are allweighted the same.
Note that I’ve brightened up the image of the flow
field a bit for benefit of the projector.

Now we apply a process called densification. Insteadof weighting all the motion vectors (or hypotheses)
equally, we look at how well they actually match foreach single pixel. It looks a bit noisy now, but it willsoon get better.

The final step is what the paper calls variationalrefinement. It essentially amounts to setting up a
nonlinear differential equation and solving it
numerically, and ends up cleaning up the field a lot.
Unfortunately, we won’t have time to go through thedetails, which is a shame, since it’s a very
fascinating and powerful technique. (If you’reinterested, I’ve made a full tutorial and put in thesource code.)
We could have stopped here if we wanted to, and justscaled up the flow field. However, we can do better.

Now we double the resolution, and start over again.
However, this time, we start our motion search atthe previous flow field. This means that eventhough the motion is now much larger compared toour block size, we have a good initial hypothesis
and still have a good shot at finding the actualmotion (now with higher precision).

Densification again.

And variational refinement. Note that the disc isstarting to show.

One more step of motion search.

Densification.

And finally, variational refinement.

0.9 ms / 1.0 ms, EPE 14.69 px

1.1 ms / 1.4 ms, EPE 13.46 px

6.1 ms / 12.9 ms, EPE 11.98 px

164.7 ms / 353.6 ms, EPE 11.75 px

720p / 1080p frame time for full interpolation, RTX 2070

So, how good is our estimated flow? This is is a test
sequence from the so-called MPI-Sintel benchmark(using Sintel from the Blender Foundation). Since
it’s synthetically rendered, we have exact motion
vectors for every pixel, and can compute theendpoint error (EPE) over the entire 50 frames.
These are the four quality presets recommended by
the paper (1–4). We can see that even at 1080p,
we are realtime at quality 3 at 60 fps; note that thisis evaluating both forward and backward flow (inparallel), computing the intermediate flow, anddoing the actual interpolation. We beat thereference code by more than an order of magnitudein performance, and a few percent in EPE.
Both visually and from the EPE numbers, we areclearly far from perfection. But is it good enough forreasonable interpolation?

This is a video of those two frames interpolated very,very slowly. You can see there are some issues
around occlusion, but overall, it’s more than decent.
Remember, at 2x, these artifacts will blur past reallyquickly.

(This slide purposefully left blank)

And here’s our test clip again, in 4x interpolation atquality 3. There are some errors if you look closely,
but overall, this is something we can live with.

Demo!

It’s time to show the actual application, and what
better way to do so than a demo. Do note that it’sreally made for 1080p—I’ve moved a few controls
and such around to fit the projector’s 1024x768.
We’re going to be running with input from file, since I
don’t have four cameras and a Nageru machinewith me. There’s twelve hours of four-camera test
data readily available from the Futatabi home page,
so you can download it and play with it yourself
without setting up a large rig.

Thank you!

https://nageru.sesse.net/

That’s it. Futatabi is part of the Nageru distribution,
since they share a fair amount of code, so that’s
also where you want to go to download it.
While we’re doing Q&A, I’m going to keep playing ahighlight reel in the background. This is a set ofreal, unedited replays made during the tournamentand rendered out at our realtime quality and speed,with mistakes and all. So it’s very much a realistic
example of what your finished video would look
like. (I’m cutting it from the PDF in the interest ofkeeping the size down a bit; you can view it onYouTube).
Thanks for listening! Or reading the slide set, asappropriate.