Good afternoon. I’m Michael Abrash, and I’m part of the group
working on virtual reality at Valve. Today I’m going to share as
much of what we’ve learned as I can cram into 25 minutes;
I’m going to go fast and cover a lot of ground, so fasten yourseat belts!

1

17 years ago, I gave a talk at GDC about the technology John
Carmack and I had developed for Quake.
That was the most fun I ever had giving a talk, because forme Quake was SF made real, literally.
You see, around 1994, I read Neal Stephenson’s Snow Crash,and instantly realized a lot of the Metaverse was doable then –and I badly wanted to be part of making it happen.
The best way I could see to do that was to join Id Software towork with John on Quake, so I did, and what we created there
actually lived up to the dream Snow Crash had put into myhead.
While it didn’t quite lead to the Metaverse – at least it hasn’t
yet – it did lead to a huge community built around realtime
networked 3D gaming, which is pretty close.
Helping to bring a whole new type of entertainment and social
interaction into existence was an amazing experience, and it
was all hugely exciting – but it’s easy to forget that Quake
actually looked like this:2And it took 15 years to get to this3

4

With that in mind, let’s come back in the present, where oneof the key missing pieces of the Metaverse – virtual reality –looks like it might be on the verge of a breakthrough.
Before we get started, I’d like to define a few terms:
Virtual reality, or VR, is when you’re immersed in a purely
virtual world.
Playing Team Fortress 2 in an Oculus Rift would be VR.
Augmented reality, or AR, is when real reality, or RR, isenhanced with virtual images that appear to coexist with thereal world.
Playing a game on a virtual chessboard that’s sitting on a realtabletop while wearing a see-through head mounted display
would be AR.
Most of what I’ll say about VR today applies to AR as well.
The key commonality between VR and AR is that virtual
images appear to exist in the same frame of reference as thereal world.
So when you move your head, virtual images have to change
correspondingly in order to appear to remain in the right5

place.
This tight association between the virtual world and the real world is
how VR and AR differ from wearable information devices such as
Google Glass. It’s far harder to keep the two worlds visually aligned
than it is to just display heads-up information.

5

Of course, we’ve heard that VR was on the verge of abreakthrough before; why should we believe it this time?
There’s no knowing for sure at this point, but it looks like thistime really may be different, due to a convergence oftechnologies, including a lot of stuff that was developed formobile but is useful for VR too.

6

There have also been tremendous advances in headmountable display technology, including projectors andwaveguides, as well as in computer vision and in hardware
that’s useful for tracking.
And finally, for the first time there’s compelling content in theform of lots of 3D games that can be ported to VR, as well asa thriving indie game community that will jump in and figureout what’s unique and fun about VR.

7

Also, this time VR isn’t just around the corner.
It’s here now, at least in the form of the Rift development kit,
and the Rift is on a path to shipping to consumers in quantity.
The Rift has a good shot at breaking through, for several
reasons:
It has a wide field of view – that is, the display covers a large
portion of the eyes’ viewing area;
It’s lightweight and ergonomic;
It’s affordable;

And there’s potentially lots of content in the form of ported 3D
games.
Most important, gaming on the Rift is highly immersive. I
remember how blown away I was the first time a rocket trail
went past me in Quake – it’s like that, but on steroids, when arocket goes past in TF2 in VR.
So the Rift has huge potential, and is very exciting.
But…

8

It’s still very early days for the Rift – this is only the firstdevelopment kit – and for VR in general, and just as was thecase with Quake, there’s lots of room for improvement in allareas. Some of those areas are familiar ones, such as:
Field of view
Mobility
Input
And resolution. The math for resolution together withwide fields of view is brutal; divide 1K by 1K resolution –about the best an affordable head mounted display islikely to be able to do in the next year – into a 100degree FOV and you get a display with less than 1/50th
the pixel density of a phone at normal viewing distance
Other areas for improvement are unique to VR and not at all
familiar, and we’ll see some of those over the remainder ofthis talk.
The bottom line is that as with 3D, it will take years, if not
decades, to fully refine VR.
AR is even harder and will take longer to make great.

9

The main point of this talk is to get you to believe this and tounderstand why it’s true, so you can make rational plans for VR
game development now and in the future.
There’s no way I can give you a proper understanding in a 25minute talk of the complexity and depth of the issues associatedwith making virtual images seem real to the human perceptual
system, but I can give you a sense of the breadth of those issues,along with a deeper look at one specific problem, and that’s what
I’m going to do today.

9It seems so simple – isn’t VR just a matter of putting a display
in a visor and showing images on it?
That actually turns out to be hard all by itself.
But solving it just gets you to…

10

The really hard parts.
There are three really hard problems that if solved would
make VR far more convincing
Tracking
Latency
And stimulating the human perceptual system toproduce results indistinguishable from the real world
All three are different aspects of the core problem, which is
the interaction between the display and the human perceptual
system
You may well wonder whether head mounted displays arereally so different from monitors.
The answer is yes – and more so than you’d imagine, for two
key reasons.

11The first reason is that, as I mentioned earlier, in order for
virtual images to seem real, they have to appear to remain in
the correct position relative to the real world at all times. Thatmeans, for example, that as the head turns in this slide, thevirtual view has to change correspondingly to show the correct
image for the new head position, just as would happen with areal-world view.
This is obviously required in AR, where virtual images coexist
with the real world, but it’s required in VR as well, eventhough you can’t see the real world, because you have a good
sense of your orientation, position, and movements even
without the help of vision, thanks to hardware and software inyour head that’s effectively a gyroscope, accelerometer, andsensor fusion filter.

12

The second reason is that, unlike monitors, VR displays move
rapidly relative to both the real world and the eyes.
In particular, they move with your head.
Your head can move very fast – ten times as fast as your eyes
can pursue moving objects in the real world when your head
isn’t moving.
Your eyes can accurately counter-rotate just as fast.
That means that if you fixate on something while you turnyour head, your eyes remain fixed with respect to the realworld, but move very quickly relative to the display – and they
can see clearly the whole time.
It’s important to understand this, because it produces a set ofartifacts that are unique to head mounted displays.

13

This slide illustrates the relative motion between an eye andthe display. Here the eye counter-rotates to remain fixated ona tree in the real world, while the head, and consequently thehead mounted display, rotates 20 degrees, something that caneasily happen in a hundred milliseconds or less. The red dot
shows the pixel on the display that maps to the tree’s
perceived location in the real world, and thus the orientationof the eye, before and after rotation, with the original position
shown with a faint red dot in the right-hand diagram. You can
see that the red dot shifts a considerable distance during therotation. While it’s true that the effect is exaggerated here
because the tree is about six inches away, it’s also true thatthe eyes and the display can move a long way relative to one
another in a short period of time.
This rapid relative motion makes it very challenging to keep
virtual images in fixed positions relative to the real world, andmatters are complicated by the fact that displays only update
once a frame.
That results in a set of major issues for VR that don’t exist in
the real world, and that are barely noticeable at worst on
monitors, because there are no useful cases in which your14

eyes move very quickly relative to a monitor while still being able tosee clearly.
Much of the rest of this talk will be dedicated to exploring some of
the implications of the relationships between a head-mounted
display, the eyes, and the real world.

14

The first implication has to do with tracking.

15

By tracking, I mean the determination of head position andorientation in the real world, known as pose.

Images have to be inexactly the right place
relative to both thehead and the realworld every frame inorder to seem real;otherwise the visual16system will detect ananomaly, no matter how
brief it is, and that will
destroy the illusion ofreality.
The human perceptual system has evolved to be very effective at
detecting such anomalies, because anomalies might be thinking
about eating you, or might be tasty.

16

In order to keep the visual system from thinking something iswrong, tracking has to be super-accurate.
How accurate?
On the order of a millimeter at 2 meters distance from thesensor.
There is currently no consumer-priced system that comesclose to the required tracking accuracy and reliability across awide enough range of motion.

17

So how close is VR tracking right now to what we really want?
Let’s look at what the tracking in the first Rift development kit
can and can’t do.
The Rift uses an inertial measurement unit, or IMU, whichcontains a gyroscope and accelerometer.
IMU-based tracking is inexpensive and lightweight, which is
good.
However, it also drifts because there’s no absolute positioning,
and it doesn’t support translation – that is, it doesn’t provide
accurate reporting of head movement from side to side, up
and down, and forward and back, and that’s a significant lack.

18

Here we see translation in action. As the head moves fromside to side, the virtual view changes accordingly, with nearer
objects shifting by greater distances than farther objects. This
produces parallax, one of the key depth cues and an
important part of making virtual scenes seem real.
In the case of the Rift, translation has little effect on thevirtual scene. I say “little” because there is a head and neckmodel that attempts to reproduce the translation of your headas it rotates, but it has no way to know about translation
resulting from any other head movement.

For those of you clever enough to notice that my illustration oftranslation actually features a Rift, I admit that the translationin these screenshots didn’t come from the head movement – itwas simulated by strafing from the keyboard. But that is howtranslation would look if the Rift did support it.
IMU tracking works for games that don’t require anything buthead rotation – FPSes, for example.
But even in FPSes, the lack of translation means you can’t
peek around corners or duck down.

In general, the lack of parallax makes virtual images seem19

less real.
Also, drift means that IMU-based VR can’t stay stable with respect
to the real world, and that rules out games like board games thatneed to stay in one place.
This is definitely a long way from where we really want to be,although bear in mind that this is only the first development kit, andOculus continues to work on tracking.

19

Next, we move on to latency.

20Latency is the delay between head motion and thecorresponding virtual world update reaching the eyes.
Too much latency results in images drawn in the right place,but at the wrong time, which can create anomalies.
The magnitude of the anomaly varies with head motion; thefaster the head moves, the greater the anomaly.
When the head reverses direction, the anomaly is briefly
multiplied and becomes far more apparent.
Again, this destroys the “reality” part of VR.

It’s also a recipe for motion sickness.
So latency needs to be super-low.

21

How low?
Somewhere between 1 & 20 ms total for the entire pipelinefrom the time head motion occurs through:Tracking
RenderingTransmitting to the display
Getting photons coming out of the display
And getting photons to stop coming out of the display
Since a single 60 Hz frame is 16.6 ms and latency in a typicalgame is 35 ms or more – often much more – it will bechallenging to get to 20 ms or less.

22

Tracking and latency are just prerequisites.
Once you have good enough tracking and latency, you candraw in the right place at the right time; then you learn about
all the other interactions of displays with the humanperceptual system.
The key here is that the way displays present photons to theeyes is nothing like the real world, and it’s a miracle we cansee anything coherent in displayed images at all.
In the real world, objects emit or reflect photons continuously.
On a display, pixels emit fixed streams of photons for discrete
periods of time.
Pixels are also fixed in space relative to the head.
This has major implications; let’s look at a couple of them.

23For the following discussion, it will be useful to understandsome very simple space-time diagrams, like the one shown
here.
The horizontal axis is x position relative to the eyes, and thevertical axis is time, advancing down the slide.
You can think of these diagrams as showing how an object oran image would move horizontally across your field of view astime passes.
You can also find a discussion of space-time diagrams on myblog; I’ll give the URL at the end of the talk.
In this diagram we have an object that is not moving relative
to the eyes. The plot is a vertical line because there’s nochange in x position over time.
It’s important to understand that the x position in these
diagrams is relative to the position and orientation of theeyes, rather than the real world, because the eyes’ frame ofreference is what matters in terms of perception. So thisdiagram could be a case where both the eyes and the object
are unmoving, or it could be a case where the eyes aresmoothly tracking an object as it moves. In either case, the24object would remain in the same x position relative to the eyes.

24This diagram shows a real-world object that’s moving from left
to right at a constant speed relative to the viewer, while theeyes remain fixated straight ahead – that is, the eyes aren’t
tracking the moving object. Here’s an example of the sort of
movement this diagram depicts.

25

26

Here’s the diagram of that movement over time again; take amoment to absorb this, because I’ll use several more of these
diagrams.
These can be trickier to interpret than you’d expect fromsomething so simple, especially if the eyes are moving, as
we’ll see later.

27

Now let’s look at the output of a display.
Here, a virtual point is moving across the screen while theeyes again remain fixated straight ahead. The point isilluminated at a single location on the display – the same pixel– for a full frame, because pixels only update once a frame. So
instead of a smooth x displacement over time, we get steppedmotion as the pixel is redrawn each frame.
Here’s an example of that stepped motion.

28

This discrete nature of photon emission over time – temporal
sampling – is key to the challenges posed by head mounted
displays.

29

One good illustration of this is color fringing.
Color-sequential liquid crystal on silicon, or LCOS, projectors
display red, green, and blue separately, one after another.
This diagram shows how the red, green, and blue components
of a moving white virtual object are displayed over time, again
with the eyes fixated straight ahead.
For a given pixel, each color displays for one-third of a frame;
because the full cycle is displayed in 16 ms, the eyes blend
the colors for that point together into a single composite color.

30

The result is that you see an image with the color properly
blended, like this. Here the three color planes are displayed
separately, one after another, and the three colored squares
line up on top of each other to produce a white square. The
red component doesn’t actually stay illuminated while thegreen and blue components display, and likewise for blue andgreen; this is just to convey the general idea of sequential
display of color components fusing to produce the final color.

31

Now, what happens if your eyes are moving relative to thedisplay, for example if you’re tracking a moving virtual object
from left to right?
The color components of a given pixel will each line up
differently with the eyes, as you can see here, and color
fringes will appear.
Remember, these diagrams are relative to the position andorientation of the eyes, not the real world.
There’s actually another important implication of this diagram,which I’ll talk about shortly.

32

Here’s how the color fringing would look – color fringes appear
at the left and right sides of the image, due to the movementof the eyes relative to the display between the times red,green, and blue are shown.
You might ask how visible the fringes can really be when awhole frame takes only 16.6 ms.
Well, if you turn your head at a leisurely speed, that’s about
100 degrees/second, believe it or not; you can easily turn atseveral hundred degrees/second.
At 120 degrees/second, 1 frame is 2 degrees.
That doesn’t sound like a lot, but two degrees can easily bedozens of pixels and that’s very noticeable.
So VR displays need to illuminate all three color components
simultaneously, or at least nearly so.
Now we that we understand a bit about the temporal sampling
done by displays, we come to persistence – that is, how longeach pixel remains lit during a frame.
If you understand why color fringing occurs, you already know

33everything you need to understand why persistence itself is aproblem.
Persistence ranges between 0 ms and an entire frame time (ormore!) for various display types.

33Remember this diagram? This is full persistence – the pixels
remain lit throughout the frame.

34

This is half persistence, where pixels remain lit for half theframe.

35

And this is zero persistence, where pixels are lit for only a tiny
fraction of the frame – but with very high intensity tocompensate for the short duration.
Both OLEDs and LCDs can be full persistence or less.
Scanning lasers are effectively zero persistence.

36

Now we come to the crux of the matter.
High persistence, together with relatively low frame rate,causes what cinematographers call “judder”, a mix ofsmearing and strobing. This diagram shows why.
Here we see the case where the eyes track a virtual object
that’s moving across the display. This could involve tracking avirtual object that appears to be moving through space, or it
could involve turning the head – to which the display is
attached – while fixating on a virtual object that isn’t moving
relative to the real world. The second case is particularly
important for two reasons: one reason is that we tend to turnto look at new things by moving our eyes first, then fixatingon the new target while the head rotates to catch up, and thesecond reason is that the relative speed between the display
and the eyes can be an order of magnitude faster when thehead rotates than when tracking a moving object without thehead turning, with correspondingly larger artifacts.
Ideally, the virtual object would stay in exactly the same
position relative to the eyes as the eyes move. However, thedisplay only updates once a frame, so, as this diagram shows,with full persistence the virtual object slides away from the37

correct location for the duration of a frame as the eyes move
relative to the display, snaps back to the right location at the startof the next frame, and then starts to slide away again.

37

We can see that effect in this video, which was shot through asee-through head-mounted display with a high-speed camera
and is being played back at one-tenth speed.
Here, the camera is panning across a wall that contains
several markers used for optical tracking, and a virtual image
is superimposed on each marker – the real-world markers are
dimly visible as patterns of black-and-white squares throughthe virtual images.
It’s easy to see that because the pixels are only updated once
per displayed frame, they slide relative to the markers for a
full displayed frame time (about 10 camera frames), thenjump back to the correct position. Exactly the same thing
happens with a full-persistence head-mounted display when
you turn your head while fixating or when you track a moving
virtual object.
Because this is being played back in slow motion, we can see
the images clearly as they move during the course of eachdisplayed frame. At real-world speeds, though, the pixel
movement is fast enough to smear across the retina, whichmakes the image blurry.

38

To give you an idea of what the smear part of judder looks
like, here's a simulation of it. The image on the left is with thehead not moving, and the image on the right is with aleisurely 120 degrees per second head turn rate. On a 60 hz
full-persistence display, that results in two degrees ofsmearing across the retina per frame – and on a head
mounted display that’s 1280 pixels and 40 degrees wide,
that's a full 64 pixels of smear, which as you can see reduces
detail considerably.
Also, at real-world speeds the jumping back to the correct
position at the start of each displayed frame makes images
strobe – that is, it causes the eyes to see multiple
simultaneous copies of each image, because they can’t fusedetailed images that move more than about five or ten arcminutes between frames.
The net result of the smearing and strobing is a loss of detailand smoothness, looking much like motion blur, whenever theeyes move relative to the display. You might think thatwouldn’t matter all that much because your eyes are moving,
but again, if you fixate on an object and turn your head, you39

can see perfectly clearly even though your eyes are moving rapidly
relative to the display, and judder will be immediately noticeable.

39

In contrast, observe how smooth the panning across text on amonitor is here.
This nicely illustrates how head mounted displays introduce a
new set of unique artifacts.
The ideal way to eliminate judder is by having a high enoughframe rate so that the eyes can’t tell the difference from thereal world; 1,000-2,000 frames per second would probably do
the trick.

40

Since that’s not a feasible option, judder can also be
eliminated with zero persistence, as shown in this diagram,where there’s no sliding or jumping at all.
We’ve done the experiment of using a zero-persistence
scanning laser display with really good tracking.
The result looks amazingly real; doing an A/B comparison witha full-persistence display is night and day.
So it would seem that zero persistence is the magic solution
for judder – but it turns out to be just another layer of theperceptual onion.
Zero persistence works perfectly for whatever image the eyes
are tracking, because that image lands in exactly the sameplace on the retina from frame to frame.

41However, as shown in this diagram, anything that is moving
rapidly relative to the eyes now strobes, because successiveframes of such images fall too far apart on the retina to fuse,
and there’s no smear to hide that.
This is a great example of how deep the issues associated withhead mounted displays go.
So at this point, judder in head mounted displays is not asolved problem.

42

Judder isn’t that bad; it does reduce visual quality and makes
text all but unreadable, and is worse with higher pixel density,
but the resulting virtual images are certainly good enough forgaming.
However, it’s not great, either, and the point here is that,much as was the case with 3D graphics, great VR visualquality is going to require a considerable amount of time andR&D.
And believe me, there’s a lot more than just color fringing andjudder to figure out.

43This slide shows additional ways in which the sampled natureof head-mounted displays may not produce the sameinteraction with the visual system that the real world does.

44

And here are yet more perceptual factors.
So is that, finally, the end of the list of challenges? Hardly.

45

After all that, we come to the really really hard problems.
Solving these is probably not required for VR to be successful,
but it probably is required for VR to be great.
I’m not going to be able to talk about most of these today, but
the single most important problem to be solved is figuring out
what the compelling experiences are that can only be had inVR.
It’ll be obvious in retrospect what those experiences are, but
it’s never obvious early on with new technology. During
development, Quake had only static lighting by way of bakedin lightmaps, and John was adamant that dynamic lighting
was not necessary. At GDC, however – the same GDC where I
talked about Quake – Billy Zelznack gave us a demo of anengine he was building, a demo that happened to include arocket that cast a dynamic light. When we got back to Dallas,
John asked me if I wanted to take a shot at implementing
dynamic lights for rockets, and when I said I had somethingelse to finish up first, he said he bet he could do it in one hour.
It actually took him only a few minutes more than that – andit was amazing how much more real the world seemed withdynamic lighting. There will be many, many such things with46

VR, and they’re all waiting for someone to find them.

46

And here are some of the really hard AR problems, which
unfortunately I don’t have time to discuss either.

47

So how might this new world of gaming come to pass?
The roadmap for VR is likely to be pretty straightforward.
First, the Rift ships and is reasonably successful, and thatkicks off the kind of positive spiral that occurred with 3D
accelerators, with several companies competing andconstantly improving the technology.
The real question is whether VR is a niche or a whole newplatform – how broadly important it turns out to be.
In the long run, it’s certainly potentially a new platform; justwatch any “Star Trek” episode that has the Holodeck, or read
Ready Player One.
It’s just not clear how long it’ll be until we can do most of
that.

48

The AR roadmap is less clear.
We know where we’d like to end up – with seamless, goanywhere, always-on AR like Rainbows End – but that’s much
too far from current technology to even think about right now.
We need to start by developing more tractable sorts of AR.
Possible roadmap #1 involves the success and widespread use
of HUD-style head mounted displays like Google Glass, whichshow information, but don’t do AR.
Once they’re established, though, AR could become a valueadded, differentiating feature - for example, allowing you toplay virtual tabletop games at home, in airports, or in boring
meetings.
In possible roadmap #2, living-room AR successfully ships fora console and becomes the dominant form of living-room
gaming.
In either case, once AR has a toehold, it can start to evolve
toward science fiction AR.
That’s definitely going to take decades to fully refine.

49

I’ve just spent 25 minutes telling you how hard VR is – andthat’s certainly true. But realtime 3D was equally hard – justcheck out Chapter 64 in my Black Book about the lengths John
went to in order to solve the potentially visible set problem, or
think about how crude the early 3D accelerators were – andover time all that has worked out amazingly well.
This is the kind of opportunity that everyone in the gaming
industry should dream of; if you want to do challenging workthat has the potential to affect almost every game written infive or ten years, VR is a great place to be right now.

It really is like when I was working on Quake - a new world is
emerging.
My guess is that the next few years will see more change in
the gaming experience than the seventeen since I gave mytalk about Quake. I could be wrong – but I hope I’m not, andI’m excited to find out!
Next, Joe Ludwig will talk about some of what we’ve learned
porting TF2 to virtual reality – and after Joe’s talk, both Joe
and I will do Q&A.

50