John Penczek*

University of Colorado

Boulder

and

Engineering Laboratory

National Institute of Standards and

Technology1

325 Broadway, MS 686.01

Boulder, CO 80305

Steven G. Sattereld

Information Technology Laboratory

National Institute of Standards and

Technology

Gaithersburg

Edward F. Kelley

KELTEK

Timothy Scheitlin

Computational and Information

Systems Laboratory

National Center for Atmospheric

Research

Judith E. Terrill

Information Technology Laboratory

National Institute of Standards and

Technology

Gaithersburg

Paul A. Boynton

Engineering Laboratory

National Institute of Standards and

Technology

Gaithersburg

Evaluating the Optical

Characteristics of Stereoscopic

Immersive Display Systems

Abstract

As large immersive displays have evolved over the years, the measurement methods

used to characterize them must also advance to keep up with the changing technologies and topologies. We propose a general suite of optical measurements that can

be used to determine the basic visual performance characteristics for a variety of

immersive display systems. These methods utilize current display industry best practices and new research that anticipates the measurement challenges posed by the

new technologies. We discuss the need for higher resolution detectors for the new

generation of laser and LED (light-emitting diode) projector systems. The introduction of multi-primary displays is addressed by the implementation of new test patterns that better simulate the display performance of typical images. Methods to

evaluate the unique attributes of stereoscopic displays, such as cross-talk and left

eye/right eye differences, are described and interpreted. In addition, it is shown that

the ambient lighting environment or display topology can have a detrimental impact

on the display image quality. The application of these measurement methodologies is

demonstrated by the evaluation of three display systems: two rear-projection and

one front-projection display. We highlight how these measurements can identify

potential display performance limitations, and offer advice on how to address some

of these limitations.

1 Introduction

Immersive displays are under considerable scrutiny as a means to enhance

the user experience and assist in our understanding of complex problems. They

are not only gaining popularity in the entertainment industry, but are also prevalent in a variety of elds as researchers try to obtain greater insight in analyzing

complicated datasets or models. Commercial and research applications range

from drug discovery, energy research, geospatial analysis, to virtual reality training scenarios. These applications often use the immersive display to leverage the

huge processing power of the human visual system to analyze the image. Display manufacturers have tried to further exploit our visual sense of spatial

immersion (to enhance a viewer’s perception of being physically present in the

scene) through improvements in the image eld of view, resolution, and the

addition of stereoscopic images. It is generally considered that an image eld of

Presence, Vol. 24, No. 4, Fall 2015, 279–297

doi:10.1162/PRES_a_00235

No rights reserved. This work was authored as part of

the Contributor’s ofcial duties as an Employee of the

United States Government and is therefore a work

of the United States Government. In accordance with

17 U.S.C. 105, no copyright protection is available for

such works under U.S. law.

*Correspondence to jpenczek@nist.gov.

1. Technology Administration, U.S. Department of Commerce. This is a contribution of the

National Institute of Standards and Technology, and is not subject to copyright.

Penczek et al. 279
view greater than 608 is necessary in order to induce a

strong sense of spatial immersion, or presence (Hale &

Stanney, 2015). Some designs pursue the immersive experience through head-mounted displays, where each

user can have a distinct view; however, this article will

focus on designs where several users can be positioned in

front of large displays and collectively share the same

physical space and visual information.

Initial efforts to improve the immersive experience

pursued larger screen sizes, which enveloped a larger

eld of view. Subsequent evolutions of immersive displays have further expanded the eld of view such that

the user could be completely contained within the visual

scene. The development of the cave automatic virtual

environment (CAVE) surrounded the viewer with three

to six large screens in a room-sized cube (Cruz-Neira,

Sandin, DeFanti, Kenyon, & Hart, 1992; Cruz-Neira,

Sandin, & DeFanti, 1993). Stereoscopic imagery and

head-tracking could also be added to enhance the visual

experience by giving the user greater perspective and

viewer position feedback. As these new immersive display

technologies are introduced, they have an impact on the

visual quality of the content that is being presented. It is

critical that the visual performance of the immersive display system be evaluated in order to understand how to

maximize the visual experience.

We propose a suite of optical measurements that characterize how well a display system renders the intended

image content to the viewer. These measurements determine some of the basic performance characteristics of

the displays, but do not capture the intangible immersive

experience that one might feel in a CAVE environment.

Even so, a knowledge of these properties can guide the

display manufacturer toward better display designs, and

advise the display system operator on more effective ways

to present their content. We rst highlight some of the

critical visual characteristics of a display system, then

present current best practice measuring methods to

determine these characteristics, and nally demonstrate

the implementation of these measurement methods on

three different immersive display systems. Initially, we

employ the characterization methods on single-screen

front- and rear-projection systems in order to determine

the intrinsic capabilities of the display system. We then

extend the measuring methods to include the impact of

the adjacent screens on the main screen, using a CAVE

system.

2 Essential Display Visual Performance

Characteristics

The display industry has developed objective visual

performance characteristics that can be used to evaluate

the image rendering capability of a display. These characteristics are generally expressed in terms of photometric

and colorimetric values, which take into account the visual response of the human eye (Berns, 2000). The eye

mainly perceives the spatial and temporal changes in

brightness and color of the observed images. We will

focus on the static characteristics of images as rendered

by immersive displays. Temporal characteristics can also

be important, but are addressed in standards published

by the International Committee for Display Metrology

(ICDM) and the International Electrotechnical Commission (IEC) (e.g., ICDM, 2012; IEC 62341-6-3, 2012).

We will briey review some of the critical display attributes, and discuss their impact on the viewing experience.

2.1 Optical Detectors

Calibrated optical detectors are used to acquire the

raw radiometric data, and convert it to absolute photometric and colorimetric data. For photometric data, the

visible spectrum is weighted by the photopic response of

the eye and expressed in terms of luminance or illuminance. The illuminance has units of lux, and describes

the amount of photopically weighted light incidence on

a surface (like a projection screen). The luminance has

units of candela per meter squared (cd/m2

), and roughly

corresponds to the relative brightness of an object. The

Commission Internationale de l’Eclaige (CIE) has standardized the human perception of color by weighting the

visible spectrum with the color-matching functions of

the CIE 1931 standard colorimetric observer (ISO

11664-1, 2007). The resulting tristimulus values represent a 3D color space, which is commonly transformed

into a 2D color mapping in the form of the CIE 1931

chromaticity diagram, or the more perceptually uniform

280 PRESENCE: VOLUME 24, NUMBER 4
color spacing of the CIE 1976 uniform chromaticity

scale (UCS) (CIE, 2004). This 2D mapping allows any

perceived color to be represented by CIE 1931 (x,y) or

CIE 1976 (u0

,v0) chromaticity coordinates.

The accuracy of the luminance and color data that

the optical detector reports is largely dependent on the

design of the instrument. The less expensive detectors

use lters to simulate the true photopic response (for

photometers) or color-matching functions (for colorimeters). In practice, it is difcult to create a lter and

detector combination that accurately matches the ideal

proles. Higher accuracy can be achieved by measuring

the actual visible spectrum with a spectrometer, then

calculating the photometric and colorimetric values

using the tabulated values of the photopic and colormatching functions (Berns, 2000). For displays that

have red, green, and blue primaries with relatively

broad spectra, the accuracy of the ltered detectors can

be comparable to the more expensive spectrometers.

However, as new displays start introducing LED and

laser light sources, the performance difference between

the ltered detectors and spectrometers becomes more

noticeable. Our unpublished work has shown that the

performance of typical ltered detectors degrades substantially as the bandwidth of the light source decreases,

and becomes unacceptable for the laser sources. For

narrower spectral bandwidth light sources, it is usually

necessary to use spectrometers or spectroradiometers.

Our results indicated that a detector bandwidth of

 5 nm was needed for luminance and color measurements of laser sources. In addition, the light produced

by the display can have other characteristics, such as

varying polarization, laser speckle, spatial nonuniformity, and temporal modulation. For example, liquid crystal display (LCD) and polarization-based stereoscopic

displays produce polarized light. This can be problematic for spectrometers, which often use polarization-dependent spectrally dispersive elements. However, if

spectral measurements are necessary, some spectroradiometers can be purchased that are relatively polarization insensitive. Therefore, when deciding which detector to use for the display measurements, one should

carefully consider the properties of the display to be

measured.

2.2 Display Photometric

Characteristics

The display luminance, color, and their uniformity

are the most basic characteristics of a display. Luminance

is a prime attribute of the visual information. Its magnitude often dictates how striking and readable the information appears, especially relative to the background

lighting. The contrast ratio between the maximum white

luminance to the black luminance is a common ergonomic metric that is closely related to the readability of

the display. It is an important metric that conveys how

easily the observer can detect small intensity differences

between image features. Because the contrast ratio of

large immersive displays is often limited by the background lighting that can wash out the black level, these

display systems continue to pursue higher white luminance technologies. However, the drive to higher luminance levels is complicated by the need to maintain uniform luminance over the entire screen. A uniform screen

appearance is more than an aesthetics issue; it helps preserve the contrast ratio over the whole image.

2.3 Spectral and Colorimetric

Characteristics

In many cases, color provides valuable information.

Expanding the range of possible colors that the display

can render (its color gamut) enables more information

to be encoded into the greater color range. Often the

image content is encoded with a certain standard color

space, such as sRGB or the International Telecommunication Union BT.709 (IEC 61966-2-1, 1999; ITU-R

BT.709-5, 2008). This content then presumes that the

display is calibrated to accurately render that color space.

An indication of a display’s color range and how well the

display reproduces the standard color space can be

obtained by measuring the chromaticity coordinates of

the primary colors (usually red, green, and blue). These

chromaticity coordinates can be plotted as a triangle on a

CIE 1931 or 1976 chromaticity diagram relative to the

triangle formed by the standard colors. Any mismatch

between the two triangles would indicate the degree to

which the image content could not be accurately rendered. A comparison between the measured white color

Penczek et al. 281
and the standard chromaticity coordinate is another reference point for characterizing the color mismatch. It

should also be noted that human vision is particularly

sensitive to inaccurate color rendering of natural scenes,

where viewers have expectations for certain color in landscapes and esh tones.

Plotting the color gamut triangle in the chromaticity

diagram is a simple visual representation of the display’s

color capability. A more quantitative measure is to calculate the color gamut area within the triangle in the CIE

1976 chromaticity diagram (ICDM, 2012). An even

better characterization of a display’s range of perceptible

colors is given by the three-dimensional CIE 1976

(L*a*b*) color space, which is typically called the

CIELAB color space (ISO 11664-4, 2008). An appropriate sampling of rendered colors can serve to dene

the outer boundary of the three-dimensional color space

for a given display. A subsequent calculation of the color

gamut volume within that boundary provides a valuable

metric for the perceptually visible span of colors that the

display is capable of producing.

Although the use of chromaticity coordinates is a convenient shorthand formalism for describing a color, in

some cases measuring the actual spectral distribution of

the light is desirable. This is particularly useful when

measuring the reection behavior of a projection screen

or the transmission characteristics of 3D glasses.

2.4 Display Uniformity and Viewing

Dependence

The basic luminance and color measurements are

usually measured perpendicular to the display center.

However, these properties are not generally the same

over the entire display, and the luminance and color difference from the center tends to increase for larger

screens. This is particularly the case for traditional projection systems. If the luminance and color nonuniformity are large, then the image quality will depend on the

screen position. This undesirable situation can result in

distorted or hidden information.

The luminance and color differences can be quantied

by measuring the display at several standard screen positions. For small displays, the detector is typically translated to each screen position and performs the measurements perpendicular to the screen. However, for large

immersive displays, the viewer is relatively close to the

screen, so the viewer would normally pivot his/her foveal

view such that it is centered on the new position. This

viewing condition can be experimentally simulated by the

vantage-point measuring method, where the detector’s

optical axis goes through the same eye location in space as

it is tilted to measure each of the screen locations (ICDM,

2012). However, the vantage-point method does have

the disadvantage of coupling in any viewing angle dependence with the screen position nonuniformity.

The viewing angle dependence of a display can be separately characterized by measuring the changes in luminance and color with varying inclination angles at the

same screen location. The variation with viewing angle

would quantify how the image characteristics at a specic

screen location would change for an observer moving

around, or how the characteristics would differ for two

observers looking at the same content.

2.5 Display Performance with Ambient

Illumination

The display’s intrinsic performance is characterized

under dark room conditions. However, in normal operation, each display system may actually be viewed with

some ambient illumination. For emissive displays,

including projector systems, any ambient illumination

will have a detrimental effect on the image quality. The

ambient light washes out colors and degrades the black

level, thereby reducing the color range and impairing

readability via glare and lower contrast. The relative

impact of the ambient light is dependent on the lighting

conguration and the technology of the display system.

Therefore, the change in the display performance with

ambient illumination needs to be uniquely evaluated for

each immersive display system. These changes are typically characterized by the relative decrease in parameters

such as contrast ratio and color gamut.

2.6 Stereoscopic Display Performance

The use of stereoscopic displays helps to enhance

the immersive viewing experience. However, it also

282 PRESENCE: VOLUME 24, NUMBER 4
introduces more design and measurement challenges.

Whereas conventional displays render the same image to

the left and right eyes, stereoscopic displays introduce

parallax between the left and right eye images. Aside

from the difference in perspective between the two

images, the rest of the content should be the same. Signicant differences in characteristics (such as luminance,

color, alignment, and magnication) between the left

and right eye images can induce visual fatigue and discomfort (Kooi & Toet, 2004; Po ̈llo ̈n et al., 2012).

Therefore, stereoscopic displays with glasses need to be

characterized by similar (monocular) photometric and

colorimetric measurements taken on conventional displays, then a comparison of the differences between the

left and right eye data.

In addition to the monocular characteristics, one key

characteristic of stereoscopic displays is how well they

can optically isolate the left and right eye images as they

are rendered to the viewer. If an image that is intended

for one of the eyes leaks to the other eye, it can cause an

offset ‘‘ghost’’ image in the unintended eye. The light

leakage is called 3D crosstalk, and is dened as the percent of light that is leaking from the unintended image

to the intended image. Figure 1 illustrates the situation

where the left eye 3D crosstalk occurs when light from

the right eye image leaks into the left image. In this

example of a stereoscopic display with glasses, the 3D

time-sequential rendering system opens the left eye shutter glass when the black image is rendered for the left

eye, and closes the left eye shutter glass when the white

image is rendered for the right eye. Any light that leaks

through the closed shutter glass of the left eye will lead

to left eye 3D crosstalk. An example of this situation

is given in Figure 2, as obtained by a photopically

weighted camera viewing display #3 through the left eye

lens. The gure shows that for these 3D glasses, the light

leakage is most severe in the diagonals, corresponding to

the display corners. All of the 3D systems in this study

exhibited similar behavior. The ability of the liquid crystal active shutter glasses to block the light from the unintended image is angle dependent, and a function of the

eye glass design.

For the left eye 3D crosstalk calculation, the viewpoint

luminance through the left eye lens is rst measured with

a full white screen for the left eye channel and a full black

screen for the right eye channel (LLWK). Then the luminance LLKW is measured with a black screen rendered to

the left eye channel and white screen to the right eye

channel. And nally, the luminance LLKK is measured

Figure 1. Illustration of 3D crosstalk for the left eye where light from

the right eye image channel leaks into the left eye image.

Figure 2. Example of 3D crosstalk from display #3 as viewed through

the left eye glass lens.

Penczek et al. 283
with a black screen for both eyes channels. The left eye

3D crosstalk is calculated by (Woods, 2012):

XL 1⁄4 ðLLKW  LLKKÞ=ðLLWK  LLKKÞ: (1)

A similar process was used for determining the right

eye 3D crosstalk. The test patterns used for this measurement, and all other measurements discussed in this

study, are available at the National Institute of Standards

and Technology website (NIST, 2015).

Other factors such as accommodation–vergence mismatch may also induce visual discomfort. This can occur

when the 3D image stereo pairs induce an eye convergence point that is far away from the focused image at

the display screen. (This mismatch is not addressed in

this study.) They are largely effected by the 3D content.

Therefore, the content providers should be aware of

exceeding recommending guidelines when creating or

displaying 3D imagery (Ukai & Howarth, 2008; Urvoy,

Barkowsky, & Le Callet, 2013).

3 Optical Measuring Methods

A variety of optical measuring methods are utilized

to extract the critical visual characteristics of immersive

displays. The photometers, colorimeters, and spectroradiometers used to measure the photometric and colorimetric values are usually the same for both immersive

and non-immersive displays. However, which instruments should be employed, how they should be used,

and the necessary metrics to evaluate these display systems can vary, depending on the specics of the technology. Regardless, the core visual parameters (such as

luminance, color, and contrast ratio) tend to be important across display technologies. In the following section, we advise the reader on the appropriate measuring

methods for determining the essential photometric and

colorimetric characteristics as part of a rigorous evaluation of a large immersive display system. We introduce a

suite of measurements that can be used to determine the

fundamental visual characteristics of most large immersive display systems. A variety of immersive displays are

used to demonstrate the general applicability of these

methods. The photometric and colorimetric measurements from these displays are presented, highlighting

the salient characteristics. Some guidance is given in

interpreting the results, and recommendations are

offered when the performance is lacking.

3.1 Description of Display System

Test Cases

The utility and exibility of the proposed optical

measuring methods is demonstrated by conducting

measurements on two rear-projection systems, and a

front-projection system. All of these display systems used

3D active shutter glasses for stereoscopic vision. A

description of the three display systems is summarized in

Table 1. Figure 3 illustrates the typical detector setup

conditions used for each display system. The detector

was aligned to simulate the eye position of a typical

viewer. For the rst and second display systems, the user

is generally standing in front of the screen, with or without head-tracking devices. We used a 170 cm (50

7@) tall

viewer to simulate the eye height, standing about half

the screen height away from the front screen, and centered horizontally on the screen. The third display system was set up in front of a U-shaped conference table.

The detector was aligned to the height at the center of

the screen, which roughly corresponded to the seated

eye height of a typical viewer, and set back to the middle

of the conference table. Although this third display system may be perceived as immersive (with > 608) only

for a viewer near the very front of the display, this system

was included in our study in order to highlight some of

the unique aspects of front-projector systems.

The typical viewing conditions dene the nominal

viewpoint location on the front screen. Most of the

measurements were performed with the room lights off.

However, some measurements were conducted with partial room lighting. In the case of the 4-wall CAVE system, the estimated impact of the light from the side

screens and oor was also evaluated.

3.2 Basic Display Optical Performance

Characteristics

These fundamental luminance and color characteristics are usually measured in the center of the screen and

under dark room conditions. In the case of the large

284 PRESENCE: VOLUME 24, NUMBER 4
immersive display tested in this study, these values are

measured at the viewer’s eye height and the horizontal

center of the front screen (the viewpoint location). The

luminance and color measurements are usually made

over a range of colors, typically including black, peak

white, and the red, green, and blue (RGB) primary colors. Several performance metrics can be determined from

these basic measurements, such as contrast ratio, color

gamut, and color gamut area. The inclusion of additional

colors, such as the cyan, magenta, and yellow secondary

colors (CMY), often helps to better describe the color

gamut capability of the display through a color gamut

volume evaluation.

Industry best practice recommends that measurement

of colors should be input-referred. For example, when a

peak red primary is to be displayed, a maximum electrical

signal is provided at the red channel video input. The

display technology, color calibration, and color management eventually dictate the nal luminance and color

rendered to the viewer. The sRGB or BT.709 color

space is often used to render images for desktop or high

denition TV applications (IEC 61966-2-1, 1999;

Figure 3. The typical detector conguration is shown for each display system measured in this study. In the case of display #2, the gure

shows the adjacent screens turned off in order to measure the intrinsic characteristics of the front screen, which is not the normal operational

scenario for a CAVE.

Table 1. General Characteristics of the 3D Immersive Display Systems Measured in This Study

Characteristic Display System #1 Display System #2 Display System #3

Install year 2004 2013 2009

Screen size 2.67 m  2.07 m 4-wall CAVE

(3.04 m  3.04 m

per wall)

3.78 m  2.13 m

Screen resolution 1280  1024 1920  1924

(2 overlapping

projectors per screen)

1920  1080

Projector 3-chip DLP 1* lamp

projector

3-chip DLP 1* lamp

projector

3-chip DLP 1* lamp

projector

Display type Rear-projection Rear-projection Front-projection

Nominal measurement

position

1.65 m high, 1.22 m

from front

1.65 m high, 1.52 m

from front

1.46 m high, 5.4 m from

front

*DLP 1 is a registered trademark of Texas Instruments.

Penczek et al. 285
ITU-R BT.709-5, 2008). The ambient room lighting,

and unintended light from adjacent screens in CAVE

displays, can also alter the observed luminance and color.

But these extrinsic effects will be discussed in a later section.

Since the spectra of the RGB primaries for all the

immersive displays under test were relatively broad, a lter-based detector would have been appropriate for the

basic measurements. However, some of our more extensive measurements required the actual spectral data.

Therefore, a precision spectroradiometer (Photo

Research PR-740)2 with a 18 aperture and 4 nm bandwidth was used for the discrete photometric and colorimetric display measurements. In addition, the availability

of area detectors with absolute luminance and accurate

color calibrations enables high resolution mapping and

analysis of the visual information. Some examples of the

utility of area detectors will also be presented, using an

SBIG ST-2000XMIUV camera with a relative photopic

lter.

All of the display characteristics are measured using

simple test patterns, where luminance and color measurements are obtained from a fully illuminated screen, or

small rectangular box, of a given display color. Since the

displays evaluated in this study use front- or rear-projection systems, our measurement procedures have their

roots in past projection standards (IEC 61947-1, 2002;

ISO 21118, 2012). However, advances in display technologies have necessitated improvements in the measuring methods (ICDM, 2012). We demonstrate some of

the new methodologies in this study and highlight the

value of their use.

All of the systems in this study were stereoscopic displays that utilized 3D glasses. Although the fusion of

images from both eyes creates part of the immersive

effect, the monocular display characteristics are still an

important component of the display quality. The assessment of the display’s monocular optical characteristics

was demonstrated by measuring the display as observed

through the left lens of the active glasses. The lens of the

active glasses was placed directly in front of the lens of

the spectroradiometer and modulated the transmitted

light at its native refresh rate. The spectroradiometer was

allowed to automatically select its optimum integration

time depending on the signal level, which was typically

several seconds. A summary of the basic optical characteristics of each display is given in Table 2. All of these

measurements were taken with the room lights off. In

addition, in the case of display system #2, only the front

screen was turned on in order to measure the performance of the screen with the least light contamination

from the adjacent oor and side screens. Although this

setup does not represent the usual operational scheme of

the CAVE, it does characterize the intrinsic screen properties absent the immersive environment contribution.

The effect of the light contamination from the adjacent

screens will be discussed later.

Display system #2 produced the largest luminance to

the viewer at the reference viewpoint. This was likely due

to the fact that it had the newest projector technology

and that it used two projectors to illuminate a single

screen. The large luminance also helped this system to

obtain the best contrast ratio. This provided the viewer

with a greater dynamic range in which to see small differences in shades where subtle features may lie. However,

the luminance and contrast ratio were not uniform for

any of the displays. The luminance was sampled at nine

screen positions (center and 10% of the screen dimension

from the corners and edges), by using the vantage-point

method. By this method, the sampled luminance

nonuniformity was found to be roughly 50% for the

front screens of all the displays. Although this level of

nonuniformity would clearly be noticeable on a typical

desktop monitor, it is actually difcult to notice by a user

in front of these screens because of the large angular subtense of the visual eld. A higher resolution representation of the screen luminance nonuniformity can be

obtained by using a photopically corrected area detector

(camera). Figure 4 shows an example of the relative

luminance map of the screen from display system #3.

Figure 5 gives the horizontal and vertical luminance pro-

le taken through the center of the relative luminance

2. Certain commercial equipment, instruments, materials, systems,

and trade names are identied in this article in order to specify or identify technologies adequately. Such identication is not intended to

imply recommendation or endorsement by the National Institute of

Standards and Technology, nor is it intended to imply that the systems

or products identied are necessarily the best available for the purpose.

286 PRESENCE: VOLUME 24, NUMBER 4
Figure 4. Relative luminance distribution across display system #3 screen. The right image is a false color representation

of the original greyscale image on the left.

Table 2. Dark Room Monocular Optical Characteristics of the Display Systems Measured in This Study Measured

Through the Left Lens of the Active 3D Glasses

Monocular Characteristics

Display System #1

(1 screen,

rear-projection)

Display System #2

(4 screens,

rear-projection)

Display System #3

(1 screen,

front-projection)

Luminance (cd/m2

) 6.5 22 19

Contrast ratio (Lwhite/Lblack) 254 1660 1255

Luminance nonuniformity

1⁄4 100%  (1-Lmin/Lmax)

45 54 46

White color (CIE 1931 chromaticity

and CCT)

(0.3015, 0.3348)

7062 K

(0.2953, 0.3425)

7305 K

(0.3304, 0.3658)

5585 K

Color gamut area (CIE 1976 diagram) 33% 28% 36%

Color gamut volume (CIELAB

volume)

1,026,100 824,720 1,066,500

White color nonuniformity (maximum

Du0

v0)

0.0063

3.6 for blue

0.0062

3 for blue

0.0039

2 for blue

White luminance change with viewing

direction

22% (at 458) 27% (at 458) 19% (at 308)

White color chromaticity shift Du0

v0

at 458

0.0018

(0.0059 for

blue)

0.0015

(0.0045 for

blue)

0.0011 at 308

(0.0013 for

green)

Transmission of glasses (photopically

weighted)

27% 32% 30%

Percent color area loss (due to eye

glasses)

7.6% 10% 4.7%

Penczek et al. 287
map. The luminance maps provide a useful visual representation of the display nonuniformity. All three of the

projection systems tested tended to have a luminance

(and contrast ratio) peak near the center of the screen,

and dropped off toward the screen edge. As a consequence of this nonuniformity, the critical features will be

most discernable near the screen center and start fading

away near the edges. The luminance maps may also be a

useful tool to better align the optical axis of the projector

and help minimize the luminance nonuniformity.

Broadband white light sources are often described by

their correlated color temperatures (CCT), which

describes the white color based on a blackbody at the

same temperature (Berns, 2000). A measurement of the

white point chromaticity and CCT is a valuable indicator

for determining the color calibration of the display system. The white color CCTs (see Table 2) for displays #1

and #2 were measured to be a bit bluish relative to the

standard sRGB or BT.709 white point of 6504 K (IEC

61966-2-1, 1999; ITU-R BT.709-5, 2008), while display #3 was too yellow. This suggests that the coded

input color signals will not be rendered correctly for

sRGB-encoded image content. The color rendering

range (or gamut) was characterized by measuring the

colors produced by the display for pure red, green, and

blue signal inputs. The results are presented in Figure 6

on a CIE 1976 UCS diagram. The range of colors that

the display can render can be visualized by the area

within the RGB triangle in the CIE 1976 chromaticity

diagram. The relative magnitude of this color gamut area

is usually expressed as a percentage of the total area contained within the spectrum locus of all visible colors

(ICDM, 2012). Although display #1 has the same 33%

color gamut area as the sRGB standard, Figure 6 con-

rms that the RGB triangle of display #1 does not completely overlap the sRGB triangle. In this case, display #1

is not capable of producing deep blue colors that may be

encoded in the intended signal. In fact, all three projection systems were not able to cover the deep blue colors

required for sRGB or BT.709 color encoding.

The CIELAB color gamut volume values given in

Table 2 were estimated by measuring the maximum

white, red, green, blue, cyan, magenta, yellow, and black

rendered colors for each display system and calculating

the volume within those boundary points (IEC 62341-

6-2, 2012; Braun & Spaulding, 2002; CIE 168, 2005).

The color gamut volumes for displays #1 and #3 are substantially larger than the sRGB value (8.2  105

), but

these values should be tempered by how much of those

volumes overlap the sRGB color space.

The screen color nonuniformity of each display was

sampled at the same nine locations used in the vantagepoint luminance nonuniformity measurement. The color

nonuniformity was evaluated by determining the white

chromaticity difference between any two locations (i and

j) on the front screen using the following CIE 1976

chromaticity difference equation:

Du0

v01⁄4



u0

i  u0j

 2

þ v0

i  v0j

 2

r

(2)

where (u0

i,v0i) and (u0j,v0j) represent the CIE 1976 chromaticity coordinates at any combination of two screen locations, with i and j 1⁄4 1 to 9, and i = j. The amount of

color nonuniformity can be characterized by the maximum Du0

v0 value on the screen. For the three display systems in our study, the maximum Du0

v0 value for a white

screen was relatively small, ranging from 0.004 to 0.006.

However, the color nonuniformity results in Table 2 also

highlight that the blue screen was found to be more than

a factor of two worse. This suggests that the color rendering will be different across the screen, and is starting

Figure 5. Relative horizontal and vertical luminance prole through

screen center of display #3. The vertical prole is from top to bottom of

the screen with increasing pixel number.

288 PRESENCE: VOLUME 24, NUMBER 4
to approach the Du0

v01⁄4 0.04 value that is considered

noticeable for nonadjacent colors (ICDM, 2012).

The viewing angle dependence of the display characteristics were determined by changing the inclination

angle of the detector’s optical axis while maintaining the

same viewpoint position at the screen center. For displays #1 and #2, the detector could swing out to a 458

viewing direction. Display #3 could be measured out to

only 308. Projection systems generally have a nonLambertian light scattering distribution transmitted or

reected off the projection screen, where the luminance

typically drops off with viewing direction. This was

observed in the displays measured in this study, with up

to a 27% reduction in white luminance at the 458 viewing direction (see Table 2) relative to viewing at normal

incidence. However, no signicant degradations in the

contrast ratios with viewing direction were measured.

Display color was found to shift with viewing direction.

The rear-projection displays tended to exhibit the largest

white color shifts. A measure of the primary colors suggested that the blue primary was the dominant factor,

with the blue primary shifting up to Du0

v01⁄4 0.0059 at

the 458 viewing direction. However, these color shifts

are not especially large. Therefore, an observation of an

image over the measured angular range would not likely

perceive a noticeable difference.

The color performance of these stereoscopic displays

can be affected by the quality of the active shutter

glasses. The spectral transmission properties of the 3D

glasses used by all three displays was obtained (see

Figure 7) by measuring the spectral radiance of a white

screen with and without the glasses. All of the displays

Figure 6. White and RGB primary colors rendered by display #1 illustrated on the CIE 1976 UCS chromaticity diagram. The colors were measured with and without the 3D glasses and compared to the standard

sRGB color gamut.

Penczek et al. 289
tested had 3D glasses with a signicant drop-off in the

blue. Figure 6 demonstrates that this drop-off reduces

the color gamut of the display in the blue region, resulting in a 5% to 10% loss in color gamut area. In addition,

by measuring the luminance of the screen with and without glasses, the eye glass transmission data in Table 2

show that about 70% of the luminance signal is lost passing through the glasses. Although about 50% of the

luminance loss is due to the time-sequential blanking of

the eye glasses needed to produce the stereoscopic effect.

These results demonstrate that the eye glasses can play

an important part in the image quality of the display.

Since the eye glasses are by far the least expensive element of the display system, there is an opportunity for

signicant gains in display performance by using better

eye glasses or stereoscopic technology.

All of the optical measurements presented thus far

used a full screen image of a single color. However, for

some newer projector technologies, especially those that

have more than the standard three RGB primaries, it has

been observed that display performance can change

depending on the test pattern (Kelley, Lang, Silverstein,

& Brill, 2009). A color-signal white methodology has

been developed to gauge the test pattern dependence

and estimate the display performance for more realistic

imagery (Kelley, Lang, Silverstein, & Brill, 2009; ICDM,

2012).

Figure 8 illustrates the sequence of test patterns used

for these measurements. The luminance (rear-projector)

or illuminance (front-projector) measurements are typically measured at the center of the nine RGB box locations. For display technologies that do not inherently exhibit luminance loading behavior, the sum of the RGB

luminance or illuminance values should add up to the

full screen white value (ICDM, 2012). However, if the

display technology does exhibit luminance loading (e.g.,

Organic Light Emitting Diode (OLED) displays or

LCDs with local dimming), then the measurement must

take into account the signal level and size of the test patterns. For color displays in which the input signals conform to a standard set of RGB digital values, any difference between the measured full screen white luminance

(or illuminance) and the sum of the RGB measurements

indicates a lack of color additivity of the color signal primaries between the test patterns. This would suggest

that the display’s color management system renders the

images differently in each case. In that case, the use of

single full screen color test patterns is not recommended,

since these test patterns would not provide an accurate

representation of the display performance for typical imagery. For the displays measured in this study, the white

luminous ux of the front-projector and the white luminance of the rear-projectors were found to be the same

for the full white screen and color-signal white method.

This check conrms that the traditional single color full

screen measurements could be used to characterize these

displays.

The open conguration of the front-projector system

(display #3) enabled the properties of the projector and

screen to be characterized separately. This allowed for a

better understanding of how each of the major components of the system contributed to the net performance

of the display system. For example, by measuring the average white illuminance at the standard nine screen locations, and multiplying by the screen area (in square

meters), we obtained an average projector luminous ux

of 1440 lumens. This value could then be compared to

the manufacturers’ stated value for the projector. How

much of the projector light is then reected by the

screen to the viewer is dependent on the scattering pro-

le of the screen. The scatter prole can be engineered

to preferentially scatter more light into the direction perpendicular to the screen plane, which is characterized by

Figure 7. Percent spectral transmittance of left lens of active glasses

used by the three display systems in this study.

290 PRESENCE: VOLUME 24, NUMBER 4
the screen gain. The screen gain is determined by measuring the luminance reected by the screen relative to

the luminance that would be reected by a perfectly

reecting diffuser (ICDM, 2012). For display #3, the

screen gain was found to be 0.92. Since most of the

viewers are within 6 308 of the display normal, an engineered screen with higher gain could have used to

improve the system’s photometric performance. The

current screen was also found to produce minimal color

shifts.

3.3 Evaluating the Inuence of Ambient

Illumination

As with all displays, the display characteristics can

be affected by external/unintended lighting. The external lighting may come from luminaires in the room. In

the case of a CAVE conguration, unintended lighting

may also come from screens adjacent to the one being

viewed. Most often, the ambient illumination is unavoidable or necessary. However, a better understanding of

how the ambient lighting impacts the display performance can guide the display owner or content provider in

minimizing the negative aspects of the light contamination. Each of the display systems in this study had its

own unique ambient light issues. For display #1, the user

would generally view the single screen in a dark room.

However, the room itself had a light-colored carpet and

white walls. Therefore, the light from the screen could

be scattered off these surrounding surfaces and wash out

the image on the screen. The 4-sided CAVE (display #2)

represents an extreme example of a concave display geometry, where light created from adjacent areas of the

display system can unintentionally illuminate the viewing

area of interest. The conference room format of display

#3 illustrates a use case were the users may need overhead lighting in order to write, but where the lighting

design can lead to light contamination on the screen.

Each display system was measured under its unique

ambient illumination environment, and its impact on

display performance was determined.

In the case of display #1, the screen faces a white wall

3.8 m away. It was observed that light emitted from the

rear-projection screen backscattered off the surrounding

surfaces and created light contamination on the screen.

This backscatter was investigated by covering the majority of the facing white wall with a black curtain. This produced a 5% reduction in the display luminance (due to

Figure 8. The nonatile trisequence patterns used by the color-signal white method.

Penczek et al. 291
the backscatter), a 1% increase in the contrast ratio (due

to a darker black level), and a 1% increase in the color

gamut area. Therefore, the backscatter from the surrounding surfaces had a minimal impact on the display

performance. This was likely due to the low display luminance and relatively large room size.

The 4-wall CAVE of display #2 represents a situation

where the scattering surfaces are in close proximity to

the front screen, in addition to emitting their own light.

In order to quantify the worst case impact of the light

contamination from the adjacent screens, the front

screen image properties were measured with the oor

and side screens emitting at their maximum white luminance. Although this is not a typical scenario for a CAVE

system, it does help to bound the impact of the light

contamination relative to the absence of that light. This

situation resulted in a front screen luminance increase of

18%, the contrast ratio collapsed to 7.3, the color gamut

area dropped to 9.7%, and the color gamut volume was

reduced to 25% of its original value. Typical imagery

would not likely produce these dramatic effects. Since

the ambient illumination was measured separately from

the front screen emission, it could be scaled to more typical luminance values for the oor and side screens, and

then used to predict the front screen characteristics. For

example, if the oor and side screens emit at only 30% of

their maximum white luminance, the front screen luminance would increase by 5%, the contrast ratio would

decrease to 21, the color gamut area drops to 19%, and

there would be a 47% loss in the color gamut volume

compared to the case with no ambient illumination.

These are center of front screen values, and will vary over

the screen.

Display designers and content providers of these

CAVE systems should recognize that the quality of the

viewing experience at a given screen location can be

impacted by the image content on the adjacent screens.

This is illustrated by the contrast ratio maps in Figure 9.

The maps were captured by digital camera images of the

front screen rendering its minimum black color state and

its maximum white state. The grayscale camera

employed a photopic lter to simulate the eye’s

response. The 2D contrast ratio map was determined by

performing a camera pixel-by-pixel ratio of the white

eld image to the black eld image. The contrast ratio

map in the left image, with the adjacent screens turned

off, resembles the luminance prole produced by the

two overlapping rear-projectors. However, the right

image (with the adjacent screens turned on) is dominated by the light contamination from the side screens

Figure 9. Relative contrast ratio false color map of display #2, with the oor and side screens turned off (left image) and

turned on (right image). Red indicates a high contrast ratio.

292 PRESENCE: VOLUME 24, NUMBER 4
and oor. This image demonstrates that the contrast ratio is the lowest adjacent to the left and right screen, and

the oor, due to the light contamination from those

planes.

For projection displays, the amount of light contamination on the front screen is dependent on the scattering

prole of the side screens and oor. The amount of side

scatter can be reduced by using engineered screens with

higher screen gain. Alternatively, direct view displays,

such as narrower viewing range LCDs or OLED displays, can be used instead of projection displays to limit

the amount of light contamination on the adjacent

screens. But the benets of enhancing the forward light

scattering/emission would need to be balanced against

any signicant reduction in the viewing angle performance and the enhanced light contamination incident on

the opposing screen (for concave geometries). Another

approach to limiting light contamination from the adjacent screens is to use darker backgrounds in the image

content when possible.

In the conference room environment of display #3,

the overhead uorescent lights were the dominant

source of light contamination on the screen. Although

the room designers were careful to use dark materials for

the reective surfaces, some of the uorescent lighting

was directly illuminating the presentation screen. The

light contamination from the uorescent lights produced a 63% increase in the screen luminance, the contrast ratio dropped to 2.7, the color gamut area dropped

to 3.3%, and the color gamut volume was reduced to 8%

of its original value. Therefore, the ambient lighting

severely impacted the viewability of the imagery. This

could be mostly mitigated by bafing the overhead lighting, and using directional or local lighting at the conference table.

3.4 Stereoscopic Display Measurements

Several stereoscopic measuring methods have been

standardized for direct view displays (IEC 62629-12-1,

2014; ICDM, 2012), but they have yet to be developed

into projection display standards. This study adapts the

direct view stereoscopic display measuring methods.

These methods focus on 3D crosstalk, and the luminance or color differences observed between the left and

right eyes. The left–right eye differences were also evaluated as a function of viewing direction and head tilt.

The 3D crosstalk was measured at the nominal viewpoint

location and the standard equally spaced nine screen

locations using the spectroradiometer. It was also measured in high resolution over the entire screen using a

photopically weighted camera. These measurements provide an indication of the display’s 3D static image performance. The evaluation of a 3D display’s dynamic

characteristics (such as icker and motion artifacts) are

still under development.

A full white test pattern was presented on the screen

for the left and right eye signal channels. The luminance

of the screen was measured through the left lens of the

3D glasses, then through the right lens. The luminance

difference between the left and right eye for all three displays was found to be only 1% or less (see Table 3). A

similar process was followed when measuring the monocular contrast ratio difference between the eyes, and

yielded differences of less than 3%. The color differences

between the left and right eyes at the user’s nominal

viewpoint were also found to be relatively small, with the

blue primary exhibiting the largest differences. Although

the human visual system is most sensitive to color

changes in the blue region, the observed chromaticity

differences measured for the display under test were too

small to be a concern. The luminance and color differences were also measured on the standard nine locations

over the screen. The worst case values are tabulated in

Table 3. These values were higher than the nominal

viewpoint values, but still found to be at acceptable

levels.

The luminance and color difference dependence on

viewing direction followed a similar procedure as the

monocular measurements, but now for both eyes. Table

3 summarizes the viewing dependence results by listing

the worst case values. For the most part, only small left–

right eye chromaticity differences were observed. However, the chromaticity difference of the blue primary for

display #2 was approaching the Du0

v01⁄4 0.004 threshold

where adjoining colors are considered discernable. This

chromaticity difference was persistently high for all viewing conditions, which suggested an issue with the 3D

Penczek et al. 293
glasses. If that was the case, it could be readily remedied

by a different choice of 3D glasses.

The 3D crosstalk performance of the displays was

measured at the nominal viewpoint on the front screen,

in a dark room, with any side screens turned off. This

viewpoint location was the foveal vision area of the observer, and corresponded to the sweetspot of the 3D

crosstalk pattern, which tended to give the lowest crosstalk values. The simulation of foveal vision is realized by

the 18 measurement eld angle of the detector. The

viewpoint 3D crosstalk values of the three display systems in our study are given for the left eye in Table 3.

The right eye 3D crosstalk measurements yielded similar

low values. The 3D crosstalk nonuniformity of the three

display systems is expressed in Table 3 as the standard

deviation of the crosstalk at the nine standard locations

on the screen using the vantage point method. As indicated by the 3D crosstalk nonuniformity data, the crosstalk levels remained relatively low over the entire screen.

The low crosstalk values are partly a consequence of the

vantage point method for measuring screen uniformity,

which tends to align the sweetspot of the 3D crosstalk

pattern to the center of the measurement area. This is

reasonable for large immersive displays, where head and

eye glass movements are necessary in order to observe

the different areas of the display. Negligible levels of 3D

crosstalk were also measured for viewing directions up to

458, and tilting the eye glasses/head sideways up to 208.

Additional information may be obtained via camera

images of the 3D crosstalk (as demonstrated in Figure

2), which offers a wider viewing eld. The camera images

can be used to create 2D crosstalk maps of the entire display (Penczek, Boynton, & Kelley, 2013). This can provide information on how the 3D crosstalk changes from

our central foveal vision toward our peripheral vision.

These crosstalk maps can be evaluated in a variety of

ways depending on the application. For example, Table

3 gives the camera average 3D crosstalk and standard

deviation over the entire screen for each the three display

screens in this study.

Table 3. Left and Right Eye Differences of the Stereoscopic Optical Characteristics for the Display Systems in This Study Measured

Through the Active 3D Glasses in a Dark Room

3D Characteristics

Display System

#1 (1 screen,

rear-projection)

Display System

#2 (4 screens,

rear-projection)

Display System

#3 (1 screen,

front-projection)

Viewpoint white luminance difference 0.8% 1.0% 1.0%

Max white luminance difference (from

9 locations)

1.9% 1.2% 1.5%

Viewpoint white chromaticity difference

(Du0

v0)

0.0006 (0.0027

for blue)

0.0013 (0.0034

for blue)

0.0008 (0.0020

for blue)

Max white chromaticity difference (from

9 locations) (Du0

v0)

0.0010 0.0022 0.0009

Max luminance difference with viewing

direction

1.5% 3.2% 1.1%

Max white chromaticity difference with

viewing direction (Du0

v0)

0.0006 (0.0006

for blue)

0.0016 (0.0034

for blue)

0.0011 (0.0018

for blue)

Left eye viewpoint 3D crosstalk 0.3% 0.4% 0.4%

Left eye 3D crosstalk vantage point

non-uniformity (Std. deviation of 9 points)

6 0.08% 6 0.03% 6 0.01%

Camera average left eye 3D crosstalk

(camera distance from screen)

1.4 6 1.3%

(3.3 m)

1.0 6 0.8%

(4.5 m)

1.4 6 1.3%

(5.4 m)

294 PRESENCE: VOLUME 24, NUMBER 4
4 Summary

We have proposed a suite of optical measuring

methods that can be used to evaluate a wide variety of

immersive display systems. These methods were applied

to front- and rear-projection displays, in single screen

and multi-screen CAVE environments. It was demonstrated how the methods can determine the dark room

monocular display characteristics, the inuence of ambient lighting and light from adjacent screens, and some

stereoscopic properties of the displays. The measurements reveal some of the intrinsic capabilities of a display, and give the user a greater understanding of how

the display is rendering the intended image content. This

information can be used by display designers to develop

better displays, and to guide display owners in operating

their displays more effectively.

Scientic applications of immersive displays often use

visualization to present complex datasets or scenes to a

researcher. In doing so, it is important for the content

provider to understand the limitations of the display.

The content provider could expend signicant effort on

image content that the display may not be capable of

rendering. If there are subtle shades or patterns that contain important information, then the display must have

the intrinsic contrast ratio in order for the viewer to perceive it. For the displays in this study, the projector light

source nonuniformity resulted in the best contrast ratios

lying near the center of the screen under dark room conditions. We also demonstrated that the contrast ratio can

be dramatically degraded, and its prole across the

screen modied, with the introduction of ambient lighting. For example, Figure 9 shows that the ambient light

contamination shifts the position of the best contrast ratio to the central top of the screen. Light contamination

can be especially problematic for multi-wall CAVE systems, where the images from the adjacent screen can

reduce the viewability at the region of interest. For single

screen displays, the ambient lighting can be somewhat

mitigated by a more careful conguration of the light

sources. However, the concave geometry of many CAVE

systems is prone to self-contamination issues. Therefore,

the image content needs more careful consideration,

such as minimizing the luminance levels around critical

features in images. The 3D crosstalk from stereoscopic

displays can also lead to reductions in the effective contrast ratio of the system, especially for higher spatial frequencies features (Penczek, Boynton, & Kelley, 2013).

The quality of the active shutter glasses can play a large

role in this case.

In addition to the image information encoded via intensity variations or shading, the information can also be

color encoded. This utilizes the human visual system’s

ability to discern color differences. Content providers

can exploit our sensitivity to color by enhancing certain

image features, such as using false color maps. However,

our ability to fully utilize color information is usually

limited by the display. We have demonstrated that all of

the displays in this study have a limited range of colors,

yet the number of possible colors approaches or exceeds

that of common color spaces, such as sRGB. We also

showed how the color range can be signicantly reduced

by light contamination, or the 3D glasses, although both

of these detractors can be dealt with if we are aware of

them. A more difcult problem occurs when the intrinsic

display color gamuts do not completely overlap the

standard sRGB color space. In this case, the encoded

colors in the image would be rendered differently on

each display. The lack of accurate color gamut mapping

to sRGB may not be so apparent for articial scenes or

synthetic images, but may be noticeable for natural

scenes and esh tones, where users have specic expectations of the proper color. This issue can be minimized by

a color calibration to the sRGB color space, but should

eventually be resolved through the development of

higher gamut displays.

As new technologies are introduced into immersive

displays, the viewer is likely to benet from an improved

visual experience. However, these advances will require a

thoughtful consideration of how these new systems

should be measured. When LED and laser sources are

used, the spectral resolution of the measurement detector needs to increase. Narrow bandwidth light sources

may also put further demands on the instrumentation in

regard to dynamic range, intensity saturation, polarization dependence, and frame synchronization. Further, as

the display’s image rendering gets more complex, the

measurement methods also need to adapt. For example,

Penczek et al. 295
display technologies that exhibit luminance loading, or

utilize features like local dimming, are sensitive to the

images being tested. Similarly, the introduction of multiprimary, or RGB and white primary displays, has led to a

plethora of color management solutions. Displays have

already developed to the point where they can change

their color management depending on the content in

the individual frames. The measurement community

continues to develop strategies, such as the color-signal

white measuring method, to address the growing display

complexity, yet further renements will be needed in

order to keep pace with the evolving display innovations.

This study highlights the current best practices for determining the basic characteristics of today’s displays.

Although these measurements provide important and

necessary metrics for display performance, they are still

inadequate to quantifying the ‘‘wow’’ factor of a quality

immersive display.

References

Berns, R. S. (2000). Billmeyer and Saltzman’s principles of color

technology (3rd ed.). New York: John Wiley & Sons.

Braun, G., & Spaulding, K. (2002). Method for evaluating the

color gamut and quantization characteristics of the outputreferred extended-gamut color encodings. Proceedings of the

IS&T/SID Tenth Color and Imaging Conference, 317–324.

CIE 15. (2004). Colorimetry. Vienna: International Commission on Illumination.

CIE 168. (2005). Criteria for the evaluation of extended-gamut

colour encodings. Vienna: International Commission on

Illumination.

Cruz-Neira, C., Sandin, D. J., DeFanti, T. A., Kenyon, R. V.,

& Hart, J. C. (1992). The CAVE: Audio visual experience

automatic virtual environment. Communications of the

ACM, 35(6), 65–72.

Cruz-Neira, C., Sandin, D. J., & DeFanti, T. A. (1993). Surround-screen projection-based virtual reality: The design

and implementation of the CAVE. Computer Graphics (Proceedings of SIGGRAPH ’93), ACM SIGGRAPH, 135–142.

Hale, K. S., & Stanney, K. M. (2015). Handbook of virtual

environments: Design, implementation, and applications

(2nd ed.). Boca Raton: CRC Press.

ICDM. (2012). Information display measurement standard,

V1.03. Campbell, California: Society for Information Display.

IEC 61966-2-1. (1999). Multimedia systems and equipment—

Colour measurement and management—Part 2-1: Colour

management—Default RGB colour space—sRGB. Geneva:

International Electrotechnical Commission.

IEC 61947-1. (2002). Electronic projection—Measurement and

documentation of key performance criteria—Part 1: Fixed

resolution projectors. Geneva: International Electrotechnical

Commission.

IEC 62341-6-2. (2012). Organic light emitting diode displays—Part 6-2: Measuring methods of visual quality and

ambient performance. (See Annex B.) Geneva: International

Electrotechnical Commission.

IEC 62341-6-3. (2012). Organic light emitting diode displays—Part 6-3: Measuring methods of image quality. Geneva: International Electrotechnical Commission.

IEC 62629-12-1. (2014). 3D Display devices—Part 12-1:

Measuring methods for stereoscopic displays using glasses—

Optical. Geneva: International Electrotechnical

Commission.

ISO 11664-1. (2007). Colorimetry—Part 1: CIE standard colorimetric observers. Geneva: International Organization for

Standardization.

ISO 11664-4. (2008). Colorimetry—Part 4: 1976 L*a*b* colour space. Geneva: International Organization for Standardization.

ISO 21118. (2012). Information technology—Ofce equipment—Information to be included in specication sheets—

Data projectors. Geneva: International Organization for

Standardization.

ITU-R BT.709-5. (2008). Parameter values for the HDTV

standards for production and international programme

exchange. Geneva: International Telecommunication

Union.

Kelley, E. F., Lang, K., Silverstein, L. D., & Brill, M. H.

(2009). A rational methodology for estimating the luminous

us based upon color primaries from digital projection displays. National Institute of Standards and Technology Internal Report 6657. Gaithersburg, MD.

Kooi, F., & Toet, A. (2004). Visual comfort of binocular and

3D displays. Displays, 25(2–3), 99–108.

NIST. (2015) Information Technology Laboratory. Immersive

display test patterns. Retrieved from http://www.nist.gov/

itl/math/hpcvg/idisplays.cfm

Penczek, J., Boynton, P. A., & Kelley, E. F. (2013). Measuring

3D crosstalk uniformity and its inuence on contrast ratio.

Journal of the Society for Information Display, 21(6),

225–230.

296 PRESENCE: VOLUME 24, NUMBER 4
Po ̈llo ̈n, M., Hakala, J., Bilcu, R., Ja ̈rvenpa ̈a ̈, T., Ha ̈kkinen, J.,

& Salmimaa, M. (2012). Color asymmetry in 3D imaging:

Inuence on the viewing experience. 3D Research, 3(3). doi:

10.1007/3DRes.02(2012)5

Ukai, K., & Howarth, P. A. (2008). Visual fatigue

caused by viewing stereoscopic motion images:

Background, theories, and observations. Displays, 29(2),

106–116.

Urvoy, M., Barkowsky, M., & Le Callet, P. (2013). How visual

fatigue and discomfort impact 3D-TV quality of experience:

A comprehensive review of technological, psychophysical,

and psychological factors. Annals of Telecommunications,

68(11–12), 641–655.

Woods, A. J. (2012). Crosstalk in stereoscopic displays:

A review. Journal of Electronic Imaging, 21(4). doi:

10.1117/1.JEI.21.4.040902

Penczek et al. 297