電子ディスプレイの人間工学シンポジウム2018

ライトフィールドディスプレイの研究動向
2018年 3月 2日
法政大学 情報科学部
小池 崇文 <takafumi (at) hosei.ac.jp>
12018 Takafumi KOIKE© 2016目次

1. ライトフィールドの基礎
2. ライトフィールドディスプレイの原理
3. ライトフィールドディスプレイの応用・最新研究事例
4. ライトフィールドディスプレイ関連企業

2© 2015, 2016 Takafumi KOIKE

関連研究の推移
Google Scholarにて，“Light Field” “Display”で検索
調査日: 2017/06/22
45
40

39

35
30
25
20

17 16

15

10

10
5
0

1

0

0

1

2

3

3

19

17

7

2

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017

注: SIDは含まれているが，IDW，IMID等は含まれていない．
3

2017 Takafumi KOIKE© 2015

1. ライトフィールドの基礎

4© 2015, 2016 Takafumi KOIKE

model information, such as depth values or image correspondences, is needed to extract the image values. Third, image generation involves only resampling, a simple linear process.

the convex hull of the object, and hence all views of an object
from outside its convex hull may be generated from a 4D lightfield. Second, if we are moving through an architectural model oran outdoor scene we are usually moving through a region of freespace; therefore, any view from inside this region, of objects outside the region, may be generated.

This representation of the light field is similar to the epipolar volumes used in computer vision [Bolles87] and to horizontalparallax-only holographic stereograms [Benton83]. An epipolar
volume is formed from an array of images created by translating acamera in equal increments in a single direction. Such a representation has recently been used to perform view interpolation[Katayama95]. A holographic stereogram is formed by exposing
a piece of film to an array of images captured by a camera moving
sideways. Halle has discussed how to set the camera aperture toproperly acquire images for holographic stereograms [Halle94],
and that theory is applicable to this work. Gavin Miller has also
recognized the potential synergy between true 3D display technologies and computer graphics algorithms [Miller95].

Plenoptic Function

The major issue in choosing a representation of the 4D
light field is how to parameterize the space of oriented lines.
There are several issues in choosing the parameterization:
Efficient calculation. The computation of the position of a linefrom its parameters should be fast. More importantly, for the
purposes of calculating new views, it should be easy to computethe line parameters given the viewing transformation and apixel location.

• Plenoptic Function: 我々が見た光を完全に記述する関数
Control over the set of lines. The space of all lines is infinite,(Adelson&Bergen, 1991)

but only a finite subset of line space is ever needed. For example, in the case of viewing an object we need only lines intersecting the convex hull of the object. Thus, there should be an
intuitive connection between the actual lines in 3-space and line
parameters.

There are several major challenges to using the light fieldapproach to view 3D scenes on a graphics workstation. First,
there is the choice of parameterization and representation of thelight field. Related to this is the choice of sampling pattern for thefield. Second, there is the issue of how to generate or acquire thelight field. Third, there is the problem of fast generation of different views. This requires that the slice representing rays through a
point be easily extracted, and that the slice be properly resampled
to avoid artifacts in the final image. Fourth, the obvious disadvantage of this approach is the large amount of data that may berequired. Intuitively one suspects that the light field is coherent
and that it may be compressed greatly. In the remaining sections
we discuss these issues and our proposed solutions.

f (x, y, z, , , , t)

• 7次元空間で定義• 位置 (3次元): (x, y, z)• 方向 (2次元): (✓, )波長 (1次元):
2. Representation•• 時間 (1次元): t

Uniform sampling. Given equally spaced samples in line
parameter space, the pattern of lines in 3-space should also be
uniform. In this sense, a uniform sampling pattern is one where
the number of lines in intervals between samples is constanteverywhere. Note that the correct measure for number of lines
is related to the form factor kernel [Sbert93].

The solution we propose is to parameterize lines by theirintersections with two planes in arbitrary position (see figure 1).
By convention, the coordinate system on the first plane is (u, v)
and on theFunction
second plane(Plenus*
is (s, t). An+oriented
line is defined by
Plenoptic
Opticの造語)
connecting
a
point
on
the
uv
plane
to
a
point
on
the st plane. In
We define the light field as the radiance at a point in a
*ラテン語で「完全な」practice
we
restrict
u,
v,
s,
and
t
to
lie
between0
and 1, and thus
given direction. Note that our definition is equivalent to thepoints
on
each
plane
are
restricted
tolie
withinaconvexquadrilatplenoptic function introduced by Adelson and Bergen [Adel5© 2015
Takafumi
KOIKE a light
eral.
We
call
this
representation
a
light
slab.
Intuitively,
son91]. The phrase light field was coined by A. Gershun in his
slab represents the beam of light entering one quadrilateral andclassic paper describing the radiometric properties of light in a
exiting another quadrilateral.
1
space [Gershun36].
McMillan and Bishop [McMillan95b] dis-

4D Light Fieldcuss the representation of 5D light fields as a set of panoramic
images at different 3D locations.

A nice feature of this representation is that one of theplanes may be placed at infinity. This is convenient since thenlines may be parameterized by a point and a direction. The latter
will prove useful for constructing light fields either from orthographic images or images with a fixed field of view. Furthermore,
if all calculations are performed using homogeneous coordinates,
the two cases may be handled at no additional cost.

However, the 5D representation may be reduced to 4D in
free space (regions free of occluders). This is a consequence ofthe fact that the radiance does not change along a line unless
blocked. 4D light fields may be interpreted as functions on thespace of oriented lines. The redundancy of the 5D representationis undesirable for two reasons: first, redundancy increases the sizeof the total dataset, and second, redundancy complicates thereconstruction of the radiance function from its samples. This
reduction in dimension has been used to simplify the representation of radiance emitted by luminaires [Levin71, Ashdown93].
For the remainder of this paper we will be only concerned with
4D light fields.

tv

L(u,v,s,t)u

1

For those familiar with Gershun’s paper, he actually uses the term light field tomean the irradiance vector as a function of position. For this reason P. Moon in a lat-s

Figure 1: The light slab representation.

er book [Moon81] uses the term photic field to denote what we call the light field.

Plenoptic Function (5次元)から次元を1つ下げたものが，4D Light Field (4次元)2
光は基本的に直進*するため，次元を下げることが可能．
*厳密には，光は無重力かつ真空中でのみ直進するが，一般的には光は直進すると近似して差し支えない．
6© 2015 Takafumi KOIKE

Light Field Displayの光線次元による分類
2D

3D

4D

y

y

5D

y

z
y

x

x
2D Image(x,y)

x

2D Image + 1D Direction(x, y, u)
ex. 1D Integral Imaging

4D Light Field(x, y, u, v)
ex. Integral Imaging

x
5D Light Field(x, y, z, u, v)
Plenoptic DIsplay?

z
y

y

x

x
3D Point Clouds (No Direction)(x, y, z)
ex. Laer Plasma

Holography(x, y, u, v)
much the same as integral imaging
with lens size = 07© 2015 Takafumi KOIKE

Light Field and Holography
Of course, very different in optics
However, almost same in 4D light field in
information space
Can’t record here!
Can’t record inside!

Can only record here!

Hologram or lens array camera
8© 2015 Takafumi KOIKE

2. ライトフィールドディスプレイ
の原理

9© 2015, 2016 Takafumi KOIKE

ライトフィールドディスプレイの2方式
レンズアレイベース• 元々のインテグラルフォトグラフィそのもの• 液晶ディスプレイとレンズアレイなどの組み合わせ• 標本化された光線を再現していると考えられる

光線(再生されるライトフィールド)レンズアレイ

液晶パネル等

複数(スタック)ディスプレイベース• 複数のパネル等をスタック状に組み合わ
せ (光の掛け算)てライトフィールドを密に再現• 画像の再現度は下がる可能性• 別名テンソルディスプレイ
10

光線 (再生されるライトフィールド)液晶パネル等© 2015,2016 Takafumi KOIKE© 2015

レンズアレイベース
インテグラルフォトグラフィ (IP)
インテグラルイメージング (II)Radiant
PyramidLens Array
LCD
Light

座標系

11

© 2015, 2016 Takafumi KOIKE

レンズアレイ方式の研究例
東芝の例

NHK放送技術研究所の例( “3D Display Research at NHK”, 3D Media Workshop, 2009 より引用 )NICTの例(c) Toshiba, Corp.
( http://www.toshiba.co.jp/about/press/2005̲04/pr1501.htm より引用 )東京大学 (土肥研)+日立製作所の例( http://mmc.nict.go.jp/people/shun/nict̲project/gCubik̲j.html より引用 )12© 2015, 2016 Takafumi KOIKE

roduced at a great
esolution degradaduced near the obhan when D ! 1.
ced at a great disution degradation,
iation of the neartor D calculated bytion of the curve ofes small when thees that the resoluobserver improves
creased.
stant from the exit
one should makea i max through theution b nyq ; i.e., D
hat D @ 1. Thereo be sufficient.

レンズアレイ方式の周波数特性• 立体像の解像度は，回折限界とナイキスト
Fig. 5. IP aperture.
周波数で決定される

XIT PUPIL

hy
corresponding eleFig.
6. Examples of the resolution of aperture-plate IP. The
H. Hoshino,
Isono, and I. Yuyama , “Analysis of resolution limitation of integral photography,” J. Opt. Soc. Am. A, Vol. 15, No. 8, pp. 2059-2065,
in Fig. 1. On
theF. Okano, H. viewing
distance and the pitch of the aperture are assumed to be
1998. より引用
on G of the spheri2 m and 2.78 mm, respectively, and the Nyquist frequency (i.e.,

!,

(10)

13
the 2-D resolution b nyq) 5 360
cpr. The gap g between© 2015,
the 2016
ap- Takafumi KOIKE
erture and the element image is 1 mm (solid curves) or 5 mm(dashed curves). Thick curves show the overall resolution. The
figure shows that a larger gap yields higher resolution.

複数(スタック)ディスプレイベース

http://web.media.mit.edu/~gordonw/TensorDisplays/ より引用
14

© 2015, 2016 Takafumi KOIKE

klight-

layer masks and backlight illumination patterns. Following Chaiet al. [2000], the spectral support for the light field reflected bya diffuse surface is the line !v = (do /dr )!x . Intersecting this
5. Weline with the spectral support for a given display provides a geoacrossmetric construction for the upper bound on the depth of field. Forr, withexample, the emitted light field spectrum for a parallax barrier or
ng beintegral imaging display is non-zero only for |!x |  1/(2 x) andsitions,
Displays:
Compressive
Field
Synthesis|!v | 1/(2v) (e.g., the Light
red boxes
shownin Figure 4), where x1].Tensor
Un表示するLight
Fieldをテンソルに分解，複and vDisplays
are the spatial
angular sampling
rates, respectively. In
sitions Multilayer
using
withandDirectional
Backlighting
practice,the spatial sampling rate x is the spacing between bar数のLCDの掛け算で実現する．
hat apDouglas Lanman
Matthew
HirschRameshRaskaror lenslets.
The
geometric
construction
yields thexGordonbarri-Wetzsteinrier slits/pinholes
MIT Media Lab
following expression for the depth of field:( 1
backfor |do |  dr xv ,2 x!⇠max(do ) =(23)
ve thatdrotherwise,2|do | v
scene,
h viewwhere v = (2dr /A)tan(↵/2) with A views and field of view ↵.
y comg fromThe geometric construction provides an upper bound on the depthof field for any tensor display architecture. Consider a two-layer
display with uniform backlighting, with the layers separated by actures,distance d and !0 = 1/(2p) denoting the maximum spatial freof view glasses-free 3D display using tensor displays. (Left) We introduce a new family of light field displays, dubbed
uration.
http://web.media.mit.edu/~gordonw/TensorDisplays/より引用
quency forlayers
each(e.g.,layer
withLCDs).
pixelRapid
pitch
p. Equation
theisedof stacks of light-attenuating
multilayer
temporalmodulation of21
thedefines
layers is exploited,sional
these
15
©
2015,
2016
Takafumi
KOIKE
backlighting,lightto allowlarge spectrum,
separations between
viewers.
left
to right:
targetd/2.
light field
view,
field
where
d
d/2
and
d
As
1 =(Right) From
2 =eyerlayer
LCD with uniform backlighting, and photograph of single LCD with directional backlighting. Layers are shown to
shown
in Figure4, a diamond-shapedthespectralograph.
The upper andlower rows
depict perspectives
seen to the left and to region
the right ofbounds
the display,respectively.
definedsupport for any two-layer display. The spatial cutoff frequency
ion.
!⇠max is again found byintersecting the line !v = (do /dr )!x with
1 Introduction
the boundary of the spectral support, yielding the following upper
isplays: a family ofcompressive
lightdepth
field of Consumer
boundon thefield forstereoscopic
any two-layer
display.
displays havebeen enabled by the introducll architectures employingレンズアレイ方式とくらべて周波数特性がa stack of timetion of high-speed
LCDs
andinexpensive
shutter glasses. Although✓◆nuating layers illuminated by uniform or diadoption is gradual, 2
theseddisplays are further supported by an excusing(i.e., any low-resolution lightfield emitter).
panding
content
stream,including theatrical
大きく改善している!
(do ) =
!0 and sports productions,(24)
field emitted by an N -layer, M -frame tensor ⇠maxinteractive
rames,
entertainment,
andstereoscopic
cameras. Manufacturd
+
2|d
|
0
th
ented by an N -order, rank-M tensor. Users are beginning to offer automultiscopic (glasses-free) 3D disund
on a unified optimization framewe introduce
plays, primarily based on the century-old concepts of parallax barrigative tensor factorization (NTF), encompassparisoners [Ives 1903] and integral imaging [Lippmann 1908]. Early prodarchitectures. This framework is the firstucts are restricted to mobile devices for which the limitations ofovidestwo layers
yer, multiframe light 8field decompositions,these methods, particularly narrow fields of view and reduced spathree layers
eartifacts
inter-observed with prior multilayer-only
single layer with backlight tial resolution, do not preclude adoption. Researchers are advancing
6with backlight a wide variety of technologies to address these limitations, spanningdecompositions;
it is also
the twofirstlayers
optimizaparambarrier
or integral imaging
ns combining multiple layers withdirectional
volumetric to holographic methods. Yet, with the widespread use of
4
fidelity.
fy the benefits and limitations of tensor dismobile devices, research must increasingly focus on solutions thataovidedprototype using modified
LCD panels and a2
preserve the thin form factors, low power consumption, and highTensor Display•

両者の周波数特性比較

cutoff (cycles/cm)•

ng backlight. Our efficient, GPU-based NTF
resolution expected from modern display technologies.
0
es interactive applications.
Throughsimula−30
−25
−20
−15
−10
−5
0
5
10
15
we show that tensor displays reveal practical
We are inspiredto(cm)
address the limitations of existing automuldistanceater depths of field, wider fields of view, andtiscopic displays by taking advantage of three emerging display
Figure 6: displays.
Comparisontechnologies:
of upper bounds
depth
of fieldfor paralompared to prior automultiscopic
multilayer onpanels,high-speed
temporalmodulation,
G.lax
Wetzstein
and
et
al.,
“Tensordisplays:
compressive
light
ﬁeld
synthesisusingmultilayer
displayswithanddirectional
backlighting.
As
early
as
1920,
Louis
Lumièrebarriers and integral imaging (red), two-layer (blue) and threedirectional
backlighting,”
ACM Transactions
onGraphics,
31(4),
2012. より引用
ds,automultiscopic
3D
displays,multilayer
stackedbacklitfilms
toapproximate
theappearanceofextended
antifiedlayer(green)displays objects
with uniform
backlighting, and single-layer
16 1920]. More recently, researchers
klighting, nonnegative
tensorfactorization[Lumièrehave reinvesti© 2016 Takafumi KOIKE© 2015,
ial frelayered displays,
considering
benefits of dynamic
LCDs(yellow) and two-layergated(orange)displayswiththedirectional
backDF
DATA
C ODE
and
optimization
of
the
layer
patterns.
Researchisdividedallel W
toEB V IDEO
lighting. The dashed blackline denotes the spatial cutoff frequencyinto
approaches using a single pair of temporally-modulated layers vs.
wickerfor each layer. Display
parameters
correspondto theprototypes
approaches
using threeor more static layers.
Withhigh-speed tem-

実装例: ソースコード
Tensor Displayのソースコードは，名古屋大学の藤井先生，高橋先生のグループにより，以下で公開されている．
http://www.fujii.nuee.nagoya-u.ac.jp/~takahasi/Research/LFDisplay/index.html

1718

2016 Takafumi KOIKE© 20152017 Takafumi KOIKE© 20152方式の違い
簡単のため2D LFを考える．
レンズアレイベース
0

1

1
✓
B 2 C
1
B C !
@ 3 A
3
4

2
4

◆

複数ディスプレイベース
0 1
1
B 2 C
B C⇥
@ 3 A
4

1

2

3

4

情報量が違う，再現性も違う
0

1
B 2
=B
@ 3
4
19

2
4
6
8

3
6
9
12

1
4
8 C
C
12 A
16
2016 Takafumi KOIKE© 2015

3. ライトフィールドディスプレイ
の応用・最新研究事例

20© 2015, 2016 Takafumi KOIKE

SIGGRAPH関連動向• 夏・冬2回の国際会議で，5〜10件程度がディスプレイや関連技術の発表• HMD/Near-Eye，LF，ホログラフィの発表が多い• 本日は，ここ数年のLF関連の研究を紹介
10

•

21

A. Maimone et al.

© 2016 Takafumi KOIKE© 2015,Focus3D
Tensor Displayをベースにし，HAR(HighAngular Resolution) backlight を導入し高画
質化．

Fig. 10. Simulated retinal blur and diffraction. Images show closeups of close and far matryoshka dolls from the light field shown in Fig. 3, and inset images
show further magnification. The larger doll is virtually positioned at 17 cm in front of the display and the rear doll at 18 cm into the display. The views of thelight field are evenly spaced over a pupil of a = 5 mm for a single eye, with the outermost views centered at the pupil edges. Compressed images reflect theconfiguration of our prototype display: 1 LCD layer in front of a HARB, a f = 31.8 cm focal length lens, a viewing distance of de = 127 cm, and native panel
resolution of 39.1 cycles/degree at this distance. Diffraction is approximated using the method described in Section 3.5 using the wavelengths red = 700
nm, green = 546 nm and blue = 435 nm. Rows: Synthetically refocused images of front doll (first rows) and rear doll (second rows). First column:
Source light field. Following three columns: Compressed version of the source light field using the decomposition algorithm described in Section 3.1.2 in thenoted configurations. High quality retinal blur can be achieved in the presence of diffraction (second column), but quality suffers if compression is too high(third column) or angular resolution is too low (fourth column).

4•

A. Maimone et al.

while the conjugate plane of the backlight is located 57 cm in frontof the screen. The decompositions use six time-multiplexed frames
and the target light field has 5⇥5 viewpoints over an eye apertureof 5 mm. As shown in the top row, multiple focal depths are still
supported. The decompositions (see Fig. 16, bottom row), however,differ from the case where the conjugate plane is in the pupil plane(see Fig. 6) – they show a flipped version of the mask patterns thatappear on the virtual layer floating in front of the other layers.

luminated by a high angular resolution (HAR) backlight. We showthat such a display can be optimized using the aforementioned
tensor display framework, albeit with a modified backlight illumination model. Section 3.2 assesses the structure of the backlightillumination and layer patterns produced by the decomposition;
6. CONCLUSION AND FUTURE WORK
this analysis reveals the source of enhanced brightness achieved
with Focus 3D over prior methods utilizing direct time-multiplexed
Future Work. The incorporation of tracking, a brighter backlight, and a higher quality lens would improve the practicality ofbacklight illumination schemes. Section 3.3 derives upper bounds
our display and allow evaluation on human subjects. Investigationon the accommodation range for both existing display architecturesof optical folding techniques, such as Wedge optics [Travis et al.
HAR Backlightand Focus 3D. Section 3.4 analyzes how the design is affected by
2013] may also allow the form factor of our current prototype todiffraction, and Section 3.5 concludes by showing the influence ofbe significantly reduced – resulting in a thin, light-efficient glassesfreeH.
3DFuchs.
display that
overcomes
many of the limitationsof current- Display.
diffraction andlightcompression
onD.ofretinalblur M.
quality.
A.Fig.
Maimone,G.prototype.
Wetzstein,
Lanman,Hirsch,R. Raskar,
Focus
3D: Compressive
Accommodation
11. field
Focus 3D
A stack
two transparent
LCDs
is mounted
ACM
Transactions
on Graphics
32 (5), LCD
2013.
より引用
on rails
in front of a Fresnel
lens with an additional
monitorbehindthe lens.
The rearmonitorand the lens form a HAR backlight.
3.1 Displays
with
HAR
Backlighting

As described above
shownatin
Figs.angles.
4 andTo7,show
Focus
consists
off-axis andviewpoints
steeperthe3D
theoretical
perof oursystem with higher
qualityilluminatedoptics, we simulateof an N -layerformancestack oflight-attenuating
panelsby a aobservermovingaround thecapable
display atofa muchwider range
high angular tracked
resolution(HAR)backlight
synthesizing
in supplementary Section A.1.

generation
devices.
Optimization
criteriaAn
canNalso
bestack
adjustedto224. Focus
Fig.
3D coordinate
system.
-layeroflight-attenuating
2017
2015
compensate for diffraction through the display layers, as©well
as Takafumi KOIKE
panelsisilluminated
by
a
high
angularresolution
backlight.
optical aberrations in the human eye [Huang et al. 2012]. Further-Each pixel on
thebacklight
layer
regionstackedof the viewer’s
pupil. We show
more,
exploring
theilluminates
combinationa smallof multiple
layers andopticalholds muchcomprisingpromise. Aasinaarbitrary
specificrefractive
embodimentofelementsa HAR backlight,
large lens and agle, planar display
backlight,
for instance,
will be reimaged
ontothethelens,
focalfollowing thebacklight
separateda distancedb behindsurface of an arbitrary refractive element; more sophisticated eldesignofTravis[1990].
A
generalized
systemisshowninements, such as parabolic lenses, or catadioptric systems will Fig.
be 7.

on is satisfied. The Bragg condition is satisfied if a probe wave
rojected on the DHOE with a wavefront identical to the refere wave of the recording process. The details of the principles of
Bragg condition are described in Supplementary Appendix A.
he Bragg condition, the wavefront of the probe wave is scattereddiffraction since a diffusing wavefront is recorded as the signal
e. The scattered wave is referred to as a reconstructed wave
se wavefront is identical to the signal wave. The amplitude ofprobe wave, which corresponds to the projected image informa, is preserved after diffraction. Thus, the image is displayed on
DHOE plane, which means the DHOE functions as a projectionen under the Bragg condition.

process described in Section 3.1.1. In the recording processes of
DHOEs 1 and 2, the incident angles of the reference waves are ✓1and ✓2 , respectively. In the reconstruction process, probe waves 1
and 2 are incident on the 2-layer DHOEs with different angles. The
23
2017
KOIKE
incident angles of probe waves 1 and 2 are ✓©1 2015
and Takafumi✓2 , whichareidentical to reference waves 1 and 2, respectively. Since DHOEs
1 and 2 have different angular selectivity broadbands, probe wave
2 is diffracted by DHOE 2 while it passes through the DHOE 1.
On the other hand, probe wave 2 is scattered only by DHOE 1.
Consequently, DHOEs 1 and 2 function as rear and front projection screens, respectively, for compressive displays. However, inpractical application, probe waves 1 and 2 are weakly diffracted by
DHOEs 2 and 1, respectively, which is referred to as a ghost effect.
The reconstructed waves from the DHOE layers are merged intoa single wave. Since the waves are from incoherent illumination
sources, it can be described using the following equation:

Additive Light Field Displays

R

Relay
Optics(a)(b)
Real object

Collimating optics

(b1)

(a)

DHOE

(1)

Itotal = I1 + I2 ,

DHOE

B

where Itotal is the intensity distribution of the merged wave, andI1 and I2 are the intensity distributions of the reconstructed waves
from DHOEs 1 and 2, respectively.

(b2)

PH

HOEをDiﬀuserとして用

𝑰𝑰𝟐𝟐
Probe wave 2

DHOE

Probe wave 1
𝑰𝑰𝟏𝟏

いた加算型のLighture 3: The reconstruction
process of DHOEs as illustrated withlayer DHOE system. The reconstruction system consists of colating and screen parts. Field
Details of thecollimating optics(a) and
Display
原理の提OE screen (b) are described. (b1) and (b2) indicate the angular
ctivity and transparency
案of the DHOE screen, respectively.

θ1

θ2

Merged wave
𝑰𝑰𝒕𝒕𝒕𝒕𝒕𝒕𝒕𝒕𝒕𝒕
Probe wave 2

Probe wave 1

DHOE 2

DHOE 1𝒛𝒛 = 𝒉𝒉𝟐𝟐𝒛𝒛 = 𝒉𝒉𝟏𝟏𝒛𝒛 = 𝟎𝟎: transmission
reconstruction system using beam projectors with a DHOE
r is shown in Fig. 3. First, white illumination from a back: scatteringt source is divided Seungjae
into threeLee,
primary
coloredbeams(red,green,
Changwon Jang, Seokil Moon, Jaebum Cho, and Byoungho Lee. 2016. Additive light field displays: realization of augmentedblue) using dichroic
mirrors.
Each coloredis modulated
realitywith holographic
optical beam
elements.
ACM Trans. Graph. Figure
35, 4, Article
60 (July2016) より引用
4: Theprincipleof the additive light field display is illus24with a stack of two DHOE layers. Eacha separate LCD panel. Then, a dichroic prism recombines andtratedlayerTakafumi
generatesin2017© 2015
KOIKE
des the three colored beams into relay optics. The combined
dependent additive light fields, which are merged into a single lightm passes through collimating optics, which convert the beam
field.
collimated plane wave. The function of the collimating optics
includes noise filtering and beam expanding. Finally, the col-3.2.2

Parameterization for Additive Light Fields

裸眼3D映画の提案
上下方向，左右方向に特性の異なるバリアを用いることで，映画館での利用を想定した裸眼3D
ディスプレイの提案
Table 1: Table of notations
m
ny
nx
i
j
(yi , zi )( y, z)Number of viewer rows in the cinema.
Number of rows displayed on the screen.
Number of columns displayed on thescreen.
Cinema row index.
Screen row index.
y, z position of the ith viewer row.
y, z displacement between two adjacentviewer rows.

yo j

m
ny
nx
i
j
(yi , zi )( y, z)yo j(⇢i,j , ⇣i )ws in the cinema.
ayed on the screen.
displayed on the

⌧
g
`p

h

viewer row.
ween two adjacentky coordinate of the j th slit on the verticalbarrier, j 2 {1, .., n}.
Table 1: barrier
Table whichof notations(⇢i,j , ⇣i )Point on the horizontal
should filter correctly the x coordinateof the rays emerging from ith row viewNumber of viewer rows in the cinema.
ers toward yo j . ⇣i is negative in our
Number ofEfrat,rows displayedon theMike
screen.
notations.
Netalee
Piotr Didyk,
Foshey, Wojciech Matusik, Anat Levin, “Cinema 3D: Large Scale Automultiscopic Display,”
Numberof columns displayedon theDistance
betweenthe vertical barrier and
ACM
Transactions
on ⌧Graphics
35(4), 2016より引用
screen.
the screen.
25
Cinema row index.
© 2016 Takafumi KOIKE© 2015,g
Spacing between two adjacent slits on thevertical barrier.
Screen row index.
`p row. Pixel size on the screen.
y, z position of the ith viewer
Horizontal distance between two consecy, z displacement between two adjacentutive angular positions of the viewer eye.
viewer rows.
k the vertical
Number of different angular images proy coordinate of the j th slit onjected by the screen.
barrier, j 2 {1, .., n}.
w
Width of a seat in the cinema.
Point on the horizontal barrier
whichshould filter correctly the x coordinate
Table
1:
Table
of
notationsthof the rays emerging from i row viewers toward yo j . ⇣i is negative in ournumber of rows in the displayed images. The horizontal parallax
notations.
barrier
locatedDistance between theverticalisbarrier
and behind the vertical one. This horizontal barrier isthe screen.
a set of slanted surfaces with nx vertical slits, whose structure is
Spacing between twoderived
adjacent below.
slits on thevertical barrier.
Pixel size on the screen.
Behind the set of two barriers, a standard screen is positioned. We
Horizontal distance between two consecdenote
distance
between the screen and the vertical barrier by⌧ .
utive angular positionsof the theviewereye.
screenhas
pixels of size `p and the k different angular images
Number of differentTheangularimagesprojected by the screen.are interlaced on the screen, such that each column of pixels belongs
tocinema.
another image in a repetitive pattern. This interlaced screen
Width of a seat in the

裸眼3D映画

slit on the vertical w
content is similar to that of a standard parallax barrier-based display,}.
with the exception that in our screen we also lose modest resolution
tal barrier which
the x coordinate
in they axisThefor reasons
thatparallax
will be explained in Sec. 2.3.1. To
number of rows in the displayed
images.
horizontal
rom ith row viewaccount
for
that,
we
pre-blurthey axisis located behind the vertical one. This horizontal barrier
is of the angular image content.
is negative in barrier
ourThroughout most of this paper we consider for simplicity an LCD

hFigure 4: Display principle: For each viewer (yi , zi ) (orange anthere exists ⇣i such that a barrier placed at that screen distancto proper disparities for viewers at distance zi . Concatenating
dependent barrier positions ⇣i , i = {1, ..., m} provides the desireof the slanted horizontal parallax barrier.

a set of slanted surfaces with nx vertical slits, whose structure isscreen, or that the interlaced screen content is displayed using backprojection. In both cases we assume that the light emerging thescreen is distributed in all directions, which is a desired feature fordjacent slits onBehindthethe set of two barriers,alla standard
screen isInpositioned.
We
regular screens.
Sec. 7 we discuss
equivalent configurations
denote the distance between thescreenandprojection.
the vertical barrier by⌧ .
2.2.1 Shaping the slanted barrier
using
front
n.
The screen has pixels of size `p and the k different angular images
tween two consecare interlaced on the screen, such that each column of pixels belongs
In order to derive the shape of the slanted horizontal parallaxof the viewer eye.
The pattern.
main ideaThis
behindour display
geometry is demonstrated in Fig. 4.
to another image in a repetitive
interlaced
screenconsider a viewer sitting at position (yi , zi ) and looking at thengular images proRays froma viewerat position(yi , zi ) are filtered by the verticalcontent is similar to that of a standard
parallax
barrier-based
display,
through a vertical barrier slit at height yo j . To produce thebarrier.
Asaresult,
for
each
row
of
viewers,
only
rays
with
a
with the exception that in our screen we also lose modest resolution
cinema.
parallax,
we (y
must
a horizontal
certainverticalangle
passthrough.
Since
we
separate
the
rays
from
Figure4:
Display
principle:
For
each viewer) (orange
and blue), parallax barrier at a di , ziplacein the y axis for reasons that will be explained in Sec. 2.3.1. To
⇣
which
is
a
functionoftherow
distance zi . Taking into a
different
rows
we
can
place
a
secondhorizontal
barrier
for
each
there
exists
⇣
such
that
a
barrieratthatscreendistanceleads
i , placediaccount for that, we pre-blur the y axis of the angular image content.
y angle zofi . rays
coming fromthis row, we should also s
proper disparities
for viewersthe
at distanceConcatenating
all rowat the screen
distance required
This second
Throughout most of this paperrow,
we consider
for simplicity
an LCDby its ztodistance.
Netalee
Efrat, Piotr
Didyk,
Mike Foshey,
Wojciech Matusik,
Anat
Levin, barrier“Cinema
3D: Large
Display,”dependent
positions⇣ , iScale= {1,Automultiscopic..., m}providesthedesiredshapebarrierverticallyabove/below
the vertical slit en
barrier
is essentiallya row-dependentd images. The
horizontal
parallax
screen,or that ACM
theinterlaced
screencontent
is 35(4),displayedusingback parallax barrier, which filters i horizontaljTransactions
onGraphics
2016より引用
of the slanted
horizontalIn
parallax ybarrier.
the
rays
from
this
row
such
that
properdisparity
isobserved.
o , by a distance we denote as ⇢i,j . That is, we denote thetical one. This
horizontal
barrierisprojection. In both cases we assume that the light emerging they, z position of the barrier as (⇢i,j , ⇣i ). Claim 1 derives a f
Fig. 4, orange
raysemerging
fromfora row at distancez
26
1 intersect thewhosestructureinisall directions,x vertical slits,
screenis distributed
which isa desiredfeature© 2016 Takafumi KOIKE
for (⇢i,j , ⇣i )©as2015,
a function of (yi , zi ). The concatenation ofhorizontal
barrier
at distance ⌧ |⇣1 | from the screen, andall regular screens. In Sec. 7 wediscuss parallax
equivalentconfigurations(⇢i,j , ⇣i ) for all viewer rows i = {1, ..., m} defines thblue rays emerging from a row at distance z22.2.1
intersectShaping
the horizontal
the slanted points
barrierusing front projection.
of the slanted barrier.
parallax barrier at a different distance ⌧|⇣2 | (note that ⇣i are
standard screen is positioned. We
negative according to our notations). Even though the rays fromIn order to derive the shape of the slanted horizontal parallax barrier
screen and The
the verticalbarrierby⌧. display geometry is demonstrated in Fig. 4.
main idea
behindourviewer 1 (orange rays) and viewer 2 (blue rays) reach the screen at
Claim 1 Let (y , zi ) denote the location of the ith viewer
consider a viewer sitting at position (yi , zi ) andi looking
at the screen
and the k different
angularimages
Rays froma viewerat position (yi , zi ) are filtered by the vertical

below.
vertical barrierderived
andLight Field Blender: Designing Optics and Rendering Methods
for See-Through and Aerial Near-Eye Display
Kazuki Otao
Yuta Itoh
Hiroyuki Osone
ライトフィールドと実体の光学重畳
University of Tsukuba
University of Tsukuba
University of Tsukuba

Pixie
Dust Technologies,
Inc.
Pixie Dust Technologies,
Inc.
Pixie
Dust Technologies, Inc.
Light Fieldkazuki.otao@pixiedusttech.com
Blender:
Designing
Opticsand Rendering
Methods
s1711428@s.tsukuba.ac.jp
soneo.tsukuba@gmail.com
for See-Through and Aerial Near-Eye Display

Kazuki Takazawa
Shunnosuke Kataoka
Yoichi Ochiai
ライトフィールドディスプレイと実像の光学重UniversityofTsukuba
National
InstituteofTechnology,
Universityof Tsukuba
Kazuki Otao
Yuta Itoh
Hiroyuki Osone

Dust Technologies, Inc.University of Tsukuba
Tokuyama College
Dust Technologies, Inc.
University ofPixie
Tsukuba
University ofPixie
Tsukubakazuki.takazawa@pixiedusttech.com
i12kataoaka@tokuyama.ac.jp
wizard@slis.tsukuba.ac.jp
Pixie Dust Technologies, Inc.
Pixie Dust Technologies, Inc.
Pixie Dust Technologies, Inc.
kazuki.otao@pixiedusttech.com
s1711428@s.tsukuba.ac.jp
soneo.tsukuba@gmail.com

畳を再帰性反射鏡を用いて実現

Kazuki Takazawa

University of Tsukuba
Pixie Dust Technologies, Inc.
kazuki.takazawa@pixiedusttech.com

Shunnosuke Kataoka

National Institute of Technology,
Tokuyama College
i12kataoaka@tokuyama.ac.jp

Aerial LightYoichi
OchiaiField Image(d)LCD Case
LCD

University of Tsukuba
Pixie Dust Technologies, Inc.
wizard@slis.tsukuba.ac.jp

deθ

Frame ofLens Array
Lens Array

Aerial LightField Image(d)
See-through background object

LCD Case
LCD

Frame of
TMD Plate
TMD PlatedeFrame ofFigure 1: (a) Concept of enabling a wide field of view, near-eye, see-through display using Transmissive
MirrorDevice. (b) Use
Lens
Array
θ
case for our HMD. The HMD consists of a LCD, a micro-lens array, and a TMD with micro-mirror array. (c)
The
image
obtainedLens Array
with the aerial light field image and the background object image. (d) Structure of our proposed HMD.

ABSTRACTCCSCONCEPTS
See-throughback-

Frame of
TMD PlateobjectIn this study, we propose a novel head-mounted display (HMD) de•groundHardware→ Displays and imagers;
sign for near-eye light field display which achieves a see-throughKazuki Otao, Yuta Itoh, Hiroyuki Osone, Kazuki Takazawa, Shunnosuke Kataoka, and Yoichi Ochiai, “Light field blender:
Figure 1: (a) Conceptof enabling
arendering
wide
of view,near-eye,see-throughdisplay
usingSIGGRAPH
Transmissive
MirrorDevice.
(b) Useand widefield andof view
forfieldaugmented
In thepast
years,KEYWORDS
designing
opticsmethods
forreality.
see-throughandaerialnear-eyedisplay,”
Asia 2017
Technical
Briefs,case for our HMD.
The
HMDelements
consistssuchof a asLCD,
a micro-lensarray, anda TMD with micro-mirror array. (c) The image obtained
many
opticalhalf-mirror,
beamsplitter,
and waveg2017.
より引用
Augmented Reality, Near-Eye Display, Light Field, Optical See-Through
27Structurewith the aerial lightandfortheobject image.
of our proposed HMD. © 2015,
uide field
were image
employedanbackground
optical see-throughdisplay.(d)Weuse a© 20172016 Takafumi KOIKE
Displaytransmissive mirror device (TMD) instead of conventional optical
ABSTRACT elements. A TMD consists of numerous micro-mirrors CCS
CONCEPTS
and is usuACM Reference Format:
In this study, we proposeanovel
head-mounted
display(HMD)
de•Hardwareand imagers;
ally used for real imaging system in the mid-air. We introduce a → Displays
Kazuki Otao, Yuta Itoh, Hiroyuki Osone, Kazuki Takazawa, Shunnosuke
sign for near-eye newlight method
field display
which
achieves
for the
TMD
plate ina see-throughorder to extend the previous
Kataoka, and Yoichi Ochiai. 2017. Light Field Blender: Designing Optics
and wide field ofnear-eye
view for display
augmented
reality.
In the past
years,
and Rendering Methods for See-Through and Aerial Near-Eye Display. In
to thesee-throughdisplay.
We achievea wideKEYWORDS
many optical elements
as half-mirror,
beamsplitter,andbywavegSANear-Eye’17 Technical
Briefs: LightSIGGRAPH
2017 Technical
Briefs, November 27–
field suchof view
and comfortableviewing
creating a Augmented
point light Reality,
Display,
Field,Asia
Optical
See-Throughuide were employed
for anopticalsee-throughWe use a is very simple
30, 2017, Bangkok, Thailand. ACM, New York, NY, USA, 4 pages. https://doi.
sourcewitha micro-lensarray.display.
Our configuration
Displayorg/10.1145/3145749.3149425
transmissive mirror(TMD)
conventional
opticalanddeviceconsistsof a instead
LCD forofthe
image source,a micro-lens array toelements. A TMDprovide
consiststheof light
numerous
micro-mirrors
andisusufield, and a TMD plate to provide the aerial
ACM image
Reference Format:
ally used for real in
imagingin We
the construct
mid-air. We
introducesee-througha
KazukiOtao, Yuta Itoh, Hiroyuki Osone, Kazuki Takazawa, Shunnosuke
front ofsystem
the eye.
a prototype
display
INTRODUCTION
new method for the
TMD platein orderHMD.
to extend
the previous
and Yoichi1Ochiai.
2017. Light Field Blender: Designing Optics
includinga fabricatedWe verify
the design ofKataoka,
our protoand
Rendering
Methodsfor
See-Throughand displays
Aerial Near-Eye
Display. Inand mixed reality
near-eye display typeto theusing
see-throughdisplay.
Weachieveawide
Near-eye,see-throughfor augmented
simulations and experiments, and further discuss theSA ’17 Technical Briefs:
SIGGRAPH
Asia 2017
Technicalfor
Briefs,Novemberfield of view and challenges
comfortableviewing abynovel
creatinga point
light
have
been recently
proposedvarioususes 27–
to the general public.
in buildingnear-eye,see-throughdisplay.
30, 2017, Bangkok, Thailand.
ACM, New York,USA, 4 pages.
https://doi.
source with a micro-lens array. Our configuration is very simple
In the commercial
field,NY,
MicrosoftHololensand Meta are available
org/10.1145/3145749.3149425
and consists of a LCD for the image source, a micro-lens array toto end users. In such a use case, an immersive experience is deprovide the light field, and a TMD plate to provide the aerial image
sired. However, there is still room for research on optical elements’17 Technical Briefs, November 27–30, 2017, Bangkok, Thailand
in front of the eye.SAWeconstruct a prototype see-through display
and presentation methods for visual information because the field© 2017 Association for Computing Machinery.
1 INTRODUCTION
including a fabricatedWeversionverifyofthe
designourhereprotoof view is still narrow. In addition to a high immersive experience,
This is HMD.
the author’s
the work.
It isofpostedfor your personal use. Not
for redistribution.
The definitiveand
Versionof Record
was published’17 Technicalsee-throughdisplaysaugmented
and scenery
mixed reality
type using simulations
and experiments,
further
discuss
the in SANear-eye,
we alsohave tofor
considerdistortingand decreasing scenery
Briefs: SIGGRAPH Asia 2017 Technical Briefs, November 27–30, 2017, Bangkok,
haveThailand,
been recentlybrightness.
proposed forusesproblem,
to the generalpublic. elements such as
challenges in building
a novel near-eye, see-through display.
To various
solve suchmany opticalhttps://doi.org/10.1145/3145749.3149425.
In the commercialtransmissive
field, Microsoft
Hololensand Meta
are available
LCD,
half-mirrors,
freeform
optics, waveguides, andto end users. In such a use case, an immersive experience is desired. However, there is still room for research on optical elements
SA ’17 Technical Briefs, November 27–30, 2017, Bangkok, Thailand
and presentation methods for visual information because the field© 2017 Association for Computing Machinery.
of view is still narrow. In addition to a high immersive experience,
This is the author’s version of the work. It is posted here for your personal use. Not
for redistribution. The definitive Version of Record was published in SA ’17 Technical
we also have to consider distorting scenery and decreasing scenery
Briefs: SIGGRAPH Asia 2017 Technical Briefs, November 27–30, 2017, Bangkok, Thailand,brightness. To solve such problem, many optical elements such as
https://doi.org/10.1145/3145749.3149425.
transmissive LCD, half-mirrors, freeform optics, waveguides, and28

TMD Plate2017 Takafumi KOIKE© 2015見えるということが起こるはずである。

るのに対しな お 、 各 Line に 隣 接 す る Line 上 に あ る ミ ラ ー 群

レンズの回に よ る 結 像 は 、少 し ず れ た 位 置 と な る た め 収 差 を 与 え 、
光源が点光源であっても、広がりをもつように見えることが予想される。

以下、本再帰性反射鏡が提案されている。

面対称結像光学素子として本素子を用いる場合には、、本論文では、マイクロミラーを用いた新しそ の 他 の 動 作 モ ー ド は 、迷 光 を 与 え る も の と な る た め 、学素子を提案する。これは、透過型の面対称抑制のための対策が必要となる。

3. 考 察2.1.2. 2 回 反 射 モ ー ド

3.1. 透 過 率

2 回反射モードにおいては、入射光は、ミラーホ

直接光、

素 子 、つ ま り 透 過 型 の 鏡（ Transmissive Mirror

直接光は、貫通穴の存在によって壁面反射を起こさールの直交して隣接する二つの内壁鏡面が形成する 2

単一のミラ

MD)で あ り 、2 次 元 に 限 定 せ ず ，3 次 元 立 体 像

ずに直接素子を透過してしまう光であるが、これにつ面コーナーリフレクターにおいて 2 回反射した後にデの違いに対実 像 と し て 得 る こ と が で き る 光 学 素 子 で あ る 。 い て は 単 に 格 子 越 し に 観 察 す る 光 と な る 。 ま た 、 多 重バ イ ス を 透 過 す る 。 2 面 コ ー ナ ー リ フ レ ク タ ー に お い[3,4]な ど が 提 案 さ れ て い る 。
面対称結像光学素子として本素子を用いる場合には、
MD の 構 造 お よ び そ の 動 作 原 理 に つ い て 説 明
反 射 モ ー ド に つ い て は 、 そ の 解 析 は 困 難 で あ り 、 こ こて は ， 2 鏡 面 の 垂 線 が 作 る 平 面 内 に お い て ， 光 は 入 射
そこで、本論文では、マイクロミラーを用いた新しそ の 他 の 動 作 モ ー ド は 、迷 光 を 与 え る も の と な る た め 、方向に戻る．また，2 鏡面に平行な軸に関しては，変に、いくつかの解析結果について示す。
では触れないこととする。
い結像光学素子を提案する。これは、透過型の面対称抑制のための対策が必要となる。

その結果を微少な2面コーナーリフレクターアレイ等を用化がないことから，デバイス面に対して面対称な経路

結 像 光 学 素 子 、つ ま り 透 過 型 の 鏡（ Transmissive Mirror

直接光は、貫通穴の存在によって壁面反射を起こさ

2.1.1.
モ ー ドず に 直 接 素 子 を 透 過 し て し ま う
を光
通でるあこ
．つ
そしてデバイス面には同方向を向
Device; TMD)で あ り 、2 次 元 に 限 定 せず ，3 1次回
元反
立射
体像
ると
がに
、な
こる
れに
、 提 案 す る TMD
の像
全を
体実
像像
をと
示し
すて
。得 る こ と が で き る本
く光
無と数なの
ー重
ルが形成されているが、各ミラー
の鏡
光節
学で
素は
子、
でミ
あラ
るー
。ホ ー
いル
て内
は壁
単の
に鏡
格面
子に
越１
し回
にだ
観け
察反
す射
る
るミ
。ラ
まー
たホ
、多
理

いた実像を写す鏡

ミつ
ラい
ーて
ホ説
ー明
ル を 通反
過射
すモ
るー
光ド
にに
つつ
いい
てて
考は
察、
をそ
行の
う解
。
ホ析
ーはル困を
2こ
回反射光は、全て光源の面対称位
以 下 、TMD の 構 造 お よ び そ の 動 作し
原て
理、
に
難透
で過
あす
りる
、こ
す る と 共 に 、 い く つ か の 解 析 結 果 にこ
つの
い場
て合
示、
す素
。子 面 に 垂 直
でな
はマ
触イ
れク
なロ
いミ
こラ
とー
とが
す全
るて
。同 じ

置を通過することから、結果的に面対称位置に結像す

方向を向いて平行に並んでいるものだけを考えることる。

に す る 。1 回 反 射 モ ー ド で の 結 像 の 様 子 を 図 ２ に 示 す 。

2. 動 作 原 理

2.1.1. 1 回 反 射 モ ー ド

図 １ に 、 提 案 す る TMD の 全 体 像 を 示 す 。

次に、同じ面対称結像光学素子である鏡との違いに

本節では、ミラーホール内壁
に明
１す
回る
だ（
け反
つの
い鏡て面説
図射
３ 参 照 ）。

(a)直 接図４

し て 、ミ ラ ー ホ ー ル を 通 過 す る 光 に つ い て 考 察 を 行 う 。
この場合、素子面に垂直なマイクロミラーが全て同じ

この解析方向を向いて平行に並んでいるものだけを考えること

に最大の透

たミラーホ

に す る 。1 回 反 射 モ ー ド で の 結 像 の 様 子 を 図 ２ に 示 す 。


図 1



基本構造

きの光の透

に入射した


は、面対称結像という鏡と同じ光学原理に基

モードが異



ため、観察

ら、実像を作ることができる結像光学素子で(a)平 面 鏡

本的な構造は、素子面に垂直に配置されたマラーから構成される、微小な２面コーナリフ
図 1 基本構造を並べたものとなっている。各２面コーナー

図３
図２(b) 透 過 鏡

平面鏡と透過鏡の違い

光を見てい

折の影響は
ミラーの 2

1 回反射モード

ターは微小であることから、光は細いビーム
い ず れ の 素 子 も 面 対 称 位 置 に 結 像 し 、物 体 A は 、素本素子は、面対称結像という鏡と同じ光学原理に基前川 聡, 仁田 功一, 的場 こ
修,の“微小2面コーナーリフレクタアレイを用いた面対称結像光学素子
動 作 を 解 析 す る に あ た り 、 反 射 す る ミ ラ ー 面 に子 反 対 側 に 鏡 像 B: 実像を結像する「鏡」,”
れて曲げられていく。
と し て 結 像 す る 。そ し 映像て鏡の場合にづきながら、実像を作ることができる結像光学素子で対して平 行 方49-52,
向 に 1 列 に 並 ぶ ミ ラ ー 群 (Line A) に よ るは 素 子 面 で 光 が 反 射 さ れ て し ま う た め 、図 ３ (a)の 様 に 、30(52),
あ る 。 基 本 的情報メディア学会技術報告
な構造は、素子面に垂直に配置 さ れ た マ 2016 より引用
反 射 と 、垂 直 方 向 に 1 列 に 並 ぶ ミ ラ ー 群29(Line B)に よ るモード
像 の 観 察 は 物 体 側 で 行 う こ©と2015,
にな
ため
、B は虚
像と
イクロミラーから構成される、微小な２面コーナリフ©る
2016
Takafumi
KOIKE
反 射 と に 分 け て 考 察 す る 。ま ず 、ミ ラ ー 群 (Line A)に よは、素子面に垂
レ直
クな
タ鏡
ー面
をを
並持
べつ
た２
も面
のコ
とー
なナ
っー
ている。各２面コーナー
図 ２ 1 回 反な
射モるー。ド
一方、提案素子の場合には、素子面に並んだミる反射であるが、これは 1 枚の細長い鏡が分割されてターアレイを形
リ成
フし
レた
クも
タの
ーで
はあ
微る
小。
で実
あ際
るに
こ製
とから、光は細いビーム
ラ ー ホ ー ル を 光 が 通 過 し て い く た め 、 図 ３ (b)の 様 に 、いると考えることができ、点光源 S からでた光は、ミ
合には、薄い金
属
プ
レ
ー
ト
に
正
方
形
の
貫
通
穴
この動作を解析するにあたり
すは
る物
ミ体
ラと
ー面
に分割されて曲げられていく。
像、
の反観射察
はに
反対側で行うことになり、B は実
ラ
ー
面
に
対
す
る
面
対
称
位
置
A
に
虚
像
を
作
る
。
その内壁を鏡面として形成することが考えら
対 し て 平 行 方 向 に 1 列 に 並 ぶ ミ ラ ー 群 (Line A) に よ る
像
と
し
て
空
中
に
結
像
す る 。ま た 、鏡 像 B を そ の 裏 側 か
次 に 、ミ ラ ー 群 (Line B)に よ る 反 射 に つ い て 考 察 す る 。
穴 を ミ ラ ー ホ ー ル と 呼 ぶ こ と に す る 。以 下 、
反 射 と 、垂 直 方 向 に 1 列 に 並 ぶ ミ ラ ー 群 (Line B)に よ る
2.1. 動 作 モード
ら
観
察
す
る
こ
と
に
な
るため、これは奥行きが反転した
これらの鏡による反射光は、全てのミラーに共通なミに形成された光学素子の動作について考察す反 射 と に 分 け て 考 察 す る 。ま ず 、ミ ラ ー 群 (Line A)に よ
本素子は、素子面に垂直な鏡面を持つ２面コーナー
ラー中心部を通るミラー面の垂線に対する線対称の位る反射であるが、これは 1 枚の細長い鏡が分割されてリフレクターアレイを形成したものである。実際に製置 を 通 る こ と か ら 、結 局いデるバとイ考スえ反る対こ側とのがBで点子は、透過型のきに、実
点像
光源 S からでた光は、ミ
作素
す子
るで
場あ
合り
に、
は光
、は
薄ミ
いラ
金ー
属ホ
プー
レートに正方形の貫通穴として結像する。すなわ
ち
、
線
分
AB
の
延
長
上
の
視
素子の反対側へ
と
透
過
し
て
い
く
。
こ
の
透
過
の
ラ ー 面 に 対 す る 面 対 称 位 置 A点に 虚 像 を 作 る 。
を開け、その内壁を鏡面として形成することが考えらか ら 観 察 し た 場 合 、点 光 源次Sにの
像ラ
はー
、虚
Aお
よび
、光の反射回数
じの
た穴
各を
動ミ
作ラ
モー
ーホ
ドー
がル
存と
在呼 ぶ こ
、ミ
群像
(Line
B)に
よ実
る反射について考察する。
れにる応。こと に す る 。以 下 、
像 B の 2 点が同時に観察
で
き
る
と
い
う
奇
妙
な
こ
と
が
ず、ミラーホー
ル
を
反
射
無
し
で
通
過
す
る
直
接
こ れ ら の 鏡 に よ る 反 射 光 は 、起
全てのミラーに共通なミ
このように形成された光学素子の動作について考察すこる。これは、瞳に入る、1 回 の 反 射 を 経
ラ光
ー線
中束
心の
部う
をち
通、
る上
ミ下
ラ方
ー向
面は
の実
垂線に対する線対称の位
るて。通 過 す る 1 回 反 射 モ ー ド 、
像 B を与え、左右方向は
Aこ
をと
与か
えら
る、結
こ と局
にデ
よバ
るイ
。ス 反 対 側 の B 点 に 実 像
射 で 通 過 す る 2 回こ反の射素モ子ーはド、、透そ過し型ての3素回
置虚
を像
通る
子以
であり、光はミラーホー
次 に 、視 点 の 移 動 に つとい
て
考
察
す
る
。視
点
を
Line
A
を経て透過する
多
重
反
射
モ
ー
ド
で
あ
る
。
2
面
し て 結 像 す る 。 す な わ ち 、 線 分 AB の 延 長 上 の 視 点
ルを通り素子の反対側へと透過していく。この透過のの方向に沿って移動するを常リフレクターと
かと
ら、
観A察点
しを
た通
場る
合光
、点
光に
源見
Sる
の 像 は 、虚 像 A お よ び 実
仕し
方て
に動
は作
、す
光る
の場
反合
射に
回は
数、
に2応回
じた各動作モードが存在ことから、点光源 S の像
ドがそれに該当
像はB、
のA2に
点定
が位
同す
時る
にか
観の
察よ
でう
きに
るという奇妙なことが起
すす
るる
。こ
まと
ずに
、な
ミる
ラ。
ーす
ホな
ーわ
ルち
を、
反射無しで通過する直接

Retinal 3D: Augmented Reality Near-Eye Display
Via Pupil-Tracked Light Field Projection on Retina

瞳位置のトラッキングと網膜照射型ディスプレイ
を用いたメガネ型ライトフィールドディスプレイ

光 モ ー ド 、1 回 の 反 射 を 経 て 通 過 す る 1 回 反 射 モ ー ド 、こる。これは、瞳に入る光線束のうち、上下方向は実2 回の反射で通過する 2 回反射モード、そして 3 回以像 B を与え、左右方向は虚像 A を与えることによる。

上の反射を経て透過する多重反射モードである。2 面

次 に 、視 点 の 移 動 に つ い て 考 察 す る 。視 点 を Line A

コーナーリフレクターとして動作する場合には、2 回の方向に沿って移動すると、A 点を通る光を常に見る

反射モードがそれに該当することになる。すなわち、ことから、点光源 S の像は、A に定位するかのように

Changwon Jang, Kiseung Bang, Seokil Moon, Jonghyun Kim, Seungjae Lee, and Byoungho Lee, “Retinal 3D: Augmented
Reality Near-Eye Display Via Pupil-Tracked Light Field Projection on Retina” SIGGRAPH Asia 2017, 2017. より引用

30

© 2017© 2015,2016 Takafumi KOIKE

度となって

するマイク

これによ

れぞれの動

るものの、

得るための

が上下方向

本素子を利

観察しなけ

向の視野角

190:4•Retinal 3D: Augmented Reality Near-Eye Display
Via Pupil-Tracked Light Field Projection on Retina
Jang, C. et al

Retinal 3D: Augmented Reality Near-Eye Display Via Pupil-Tracked

Fig. 7. Pictures of Retinal
Fig. 5. Conceptual diagram of focus cues generation
with lighCenter) The multi-LSP pr
module.a point source
human eye. Shadowed area indicate light cone fromthe human eye (natural blurring case) and solid red lines indicate
light rays consisting light field. When eye focuses on the point,
there are noteworthy di
are focused at a same location in retina as welltype(le�).
Wheneye disuper
multi-view
as a pupil-tracking
scheelsewhere, light rays are not focused but produces
blurred image

be signi�cantly reduced
information for a pupil
displays generate light �Fig. 6. Schematic of light field generation of Retinal 3D. Light field confor the eye-box
or mo
box
has beena on
crucialwhich hinderedretinalprsists of multiple
viewpoints
is formed
the pupildrawback,location. Each Element,
quality or computation(n)
Im , indicatesdisplay
a light ray generatedbyasinglescanofMEMS
mirror,andsystemsfrom
being
practicallyused.
�erefore,eimagestogeneratefocu(n)
I {1, 2, . . ., M } provides the n t h viewpoint, where M is the number of light rays
optimization algorithm
thea single
eye-boxretinalprojection
generated withinframe ofof
the the
LSP. An
optical pathis unfolded for view display withou
proposed scheme has a l
simplicity. The
formationof multiple
viewpoints
can be achieved
either
FOV
would
be an
importantbreak-through.
In thisperspeis muchsmaller,which
by the temporal multiplexing of a single LSP synchronized with a steering�ese concepts
proposeda novelsolution by shi�ing the positionof aretheexplemirror (top), orby using multiple
LSPs (bo�om).

Retinal 3D: Augmented Reality Near-Eye Display Via Pupil-Tracked Light Field Projection on Retina

•

190:5

by taking advantage of the optomechanicalscanning de
4 IMPLEMENTATIO

emanated froman imagetolerance
point as shown
Fig. 5.of
In thegeneral,it re- When the convergi
angularrange
HOE.

Based on the proposed cof Retinal
3D prototypes.
Le�) The single
prototype,Fig. 3. The configuration of Retinal 3D. Each red line shows beam pathFig.
of7. Picturesquireslarge amountof informationor computation
loadLSPto generateof a Retinal
3D prototy
Center) The multi-LSP
prototype,asandaRight)
The pupil-tracking
cameraisrecordedsignalbeam,
theincidenceof tilted
pro
the
light
�eld
over
a
large
eye-box
area.
In
this
study,
we
producea scanned
laserJang,
beam
whichBang,fills green
structure
ofSeungjae
laser beam
Changwon
Kiseung
Seokilarea.
Moon,The
Jonghyun
Kim,
Lee, and Byoungho Lee, “Retinal 3D: Augmented
mented to show the fea
module.

the light �eldresultsby forming
view-points
only inside
inmultiple
a lateralshi�ingof thethepupil,focus point
ofHMD.
a recona compact
Secon
Reality
Near-Eye
Viaas
Pupil-Tracked
Light Field Projection
on Retina”
SIGGRAPH
Asia
2017,
2017.
より引用
projector
partwill beDisplay
indicated
LSP for simplification.
Only a monocularrather than whole eye-box area. �is localized light �eld is cooperto demonstrate
perf
wave, as shown in Fig. 4. By
actively controlling
thistheshi�
part is depicted in the Figure.
31
ated with dynamic eye-box to provide focus cuese�ciently.
© 2017 Takafumi KOIKE© 2015, 2016 Takafumi KOIKE
frame rate. Although th
thereAsarenoteworthy
comparedwith conventional
panel
point
to
track
the movement
oflightthepupil within realtime, weshown
in Fig. di�erences
6, eachviewpointsamplesa singleraya smaller size and
type
super
First,
sincethethepupilsystemamong
themulti-view
entire lightdisplays.
bundle thatentersfromisadesigned
virtualachievewhenusingeye
the
vide
room
for
the
eye
to
move
around.
We
refer
to
this
point. Whenscheme,
the observer’s
eye focusesat theinformation
virtual image
asimagea pupil-tracking
the amountof required
canoptical path con�gurati
expansion
methodas“dynamicthese box
light
rays
are
projected
at the same
positionon theeye-box”, di�erentia
bepoint,
signi�cantly
reduced.
�esystemproposed
here
only provides
retinal plane.
Otherwise,
projected
lightsize.
rays areat
informationfora pupil-sizedarea, while
other
types
offocused
light
�eld
physical
exit-pupil
Innot
the
prototype,
weDisplayused aSystem
fast
4.1
the samepositionwhich
blurringof theimage. large
Multiple
displaysgeneratelight�eldinduces
information
withinrelatively
area
HardwareFo
Fig. 6. Schematic of light field generation of Retinal 3D. Light field conmirror
to bymodulatethephasemismatchof theDesign.
incidefocusing
pointscanbe (M2)
generatedusing
thewillsame
principle
usedforthe eye-box
or motionparallax,
which
trade
the imagesists of multiple viewpoints is formed on the pupil location. Each Element,
ized laser scanning prforthedynamic
eye-box.
To
modulatetheincidence
angle
of
a
quality or computationload. InGiven
our design,use HIC
only three
on the HIC.
thatwethehas view
focal lengthof h, the fo(n)sources. �e optical eng
Im , indicates a light ray generated by a single scan of MEMS mirror, andprobe beam,
the fastfocus
steering(FSM)
a dynamic
eyeimages
to generatecues,mirror
without
theusedneedforfor
an additional(n)tery part. For M2, a fastis determined
as follows:I {1, 2, . . ., M } provides the n t h viewpoint, where M is the number of light rays
box
can
be
utilizedwiththetemporalmultiplexing,
as
shown
at
optimization algorithm. Second, since eachin the✓ generated✓mirrorray◆ ◆ sig
was used. �e input
generated within a single frame of the LSP. An optical path is unfolded forthe top of
Fig. 6.has�escanningof a MEMSretinal
decides
proposedschemea large
DOF,speed
the unintended
blur
e�ect
1
1 + anvia asDAQ board (PCI-6
simplicity. The formation of multiple viewpoints can be achieved either
the
total
amount
of
light
�eld
informationsinceasinglestepofs = ahlarge
tan depth
sin expressible
sin p range.
sinwasracquired,
is much smaller, which ensuresby the temporal multiplexing of a single LSP synchronized with a steering
the MEMS mirror corresponds to a single light ray. To generate1 + alformationp�ese concepts are explained in more detail in Section 6.2.
(Pupil Labs). When test
mirror (top), or by using multiple LSPs (bo�om).
a su�cient number of viewpoints at a su�cient frame rate, thehead/camera wer
device performancehigh enough. Instead,
LSPsal is user’s
whereshould
an isberefractive
indexmultiplechange,lateralshrinkage
4 IMPLEMENTATION
pupil-tracking camera
can be used to enhance the bandwidth, as shown at the bo�om of
emanated from an image point as shown Fig. 5. In general, it reare
wavelengths
oftherecordedandprobe
beams,
inorder
to
not disturb
p
p an
Based
on
the
proposedconcepts,
wedesigned
and
built
two
versions
Fig. 6. �ree laser scanning projectors are aligned with slightly
Fig. 4. Dynamic
generation
by tilted
probe beamquireseye-box
large amountof informationor computationload toincident
generate on HIC.
separatelybeam
at theofdi�erent
a Retinal
3D
prototype.
amismatch,single LSP
prototypewas
imple- and�xed
anglestoincidence
induce aFirst,
phaseangles
which
formedmultiple
the
of
probe
the
beam
referenceWith di�erent
incidence
angle
of
probe
beams
and
,
signalbeams
the light �eld over a large eye-box area. In thisp1study, we
p2produce
enough to be easily inte
mentedto show
the afeasibilityof an integrated optical system asviewpoints
without
temporal multiplexing.
tively.
When wegenerate a shi�ed focus pointby inducing
thedi�raction
light �eld by forming
view-pointsforming
only insideshi�edthe pupil,focusingwith di�erent
anglesmultiple
are generated,
Speci�cationsof optica
a compact HMD. Second, a multi-LSP prototype was implemented
rather than whole eye-box area. �is localized light �eld is cooperMerits. �econceptof focuscue
generation
issystembased onsuper
beam-shaping
mismatch,di�ractione�ciencydrop
also beconditio
consi
points.
to demonstrate
the performanceof theproposedwithhighshouldated with dynamic eye-box to provide focus cues e�ciently.
multi-view
displays orthe
light�eldLSP
displays.
However,in be
Retinal3D,
single projector prototy
frame
rate. Although
singleprototype
could
designedthe di�ractione�ciency
is reduced
too much, uniform br
As shown in Fig. 6, each viewpoint samples a single light ray
with a smaller size and a lower cost, a full framerate is di�cult toamong the entire light bundle that enters the pupil from a virtual
be achieved.
maximumshi�shouldVol.b3
on Graphics,
achieve whenmay
usingnotthe current
LSP model.�erefore
Both systemsthe
havetheACM Transactions
image point. When the observer’s eye focuses at the virtual image
optical path con�guration
depictedin Fig. 3.of angular tolerance (see Section B.
insidethebandwidth
light with sometolerance,butatadi�erentdi�raction
angle
and
point, these light rays are projected at the same position on thematerial). When the maximum shi� is proe�ciency. �isaddressed
as phasemismatchof k-vector.
In Displaysupplemental
retinale�ect
plane. is
Otherwise,
projected
light rays
are not focusedat
4.1
System Implementation
the sameposition
inducesblurring toof theimage. theMultiplesmax ,Forthe�naldynamicweeye-box
size E can be determined
this study, we
inducethewhichphase
mismatch
controldi�ractionHardware Design.
theimplementation,used commercialfocusing points can be generated by using the same principle usedized
angle by modulating
theeye-box.
probe Tobeam
angle,
that itangle
canofprovide
a laser scanning projectors (Picopro, Celluon) as the display
for the dynamic
modulate
the soincidencea
E casing⇡ smax+ D,sources. �e optical engine was detached from theand batprobe
beam,cues,
the fastwhichsteering will
mirrorbe(FSM)
used for a dynamic
eye-box andfocus
explained
in theeyefollowing
tery
part.
For
M2,
a
fast
steering
mirror(OIM5002,
OpticsIn
Motion)box can be utilized with the temporal multiplexing, as shown at
will forallow
movementof thepupil, with D denoting t
sections. �edegree of di�raction angle and e�ciency changewas
is used. �ewhich
input signal
the steering
mirror was
generated
the top of Fig. 6. �e scanning speed of a MEMS mirror decides
via a DAQ board(PCI-6221,�eNI) inintensity
real-time. �eposition in-image can be re-no
diameter.
ofpupila displayed
primarily dependent
on ofthelight
characteristics
a photopolymer
such
the total amount�eld information ofsince
a single step offormation was acquired by integrating small pupil-tracking devices
the refractive
MEMS mirrorindex
corresponds
to a single lightray. Toconsidering
the di�raction e�ciency to provide uniform br
as thickness,
modulation,
degreeofgenerateshrinkage,(Pupil
or Labs). Whentested by a user or a camera, the prototype anda su�cient number of viewpoints at a su�cient frame rate, theMoreweredetailed
ofholders.
a dynamic
eye-box are desuser’s head/camera�xed oncharacteristics
the optical table withA
con�guration
ofperformancethe HOE.should
Sincebeweadvantageof theproperty,
devicehightakeenough.
Instead, multiple
LSPs
pupil-tracking
camera module wasSection
located o�-axis
supplementary
C. from the eye
canasbeangularused to enhance
the bandwidth,
at theselectivity.
bo�om ofwe refer to it
tolerance
insteadasofshown
angularGenin order to not disturb the observer’s sight. �e pupil-tracker wasFig. 6. �ree laser scanning projectors are aligned with slightly
erally, re�ection
type HOE with thin thickness has large angular�xed separately at the bo�om of the HIC, but the size was smalldi�erent angles to induce a phase mismatch, which formed multiple
Focus within
CuetheGeneration
via
the Localized Light
enough to be 3.4
easily integrated
frame of a pair ofglasses.
tolerance, which
is without
suitablefor themultiplexing.
purpose.
viewpoints
a temporal
Speci�cations of optical components were designed to satisfy the
Scanning
Merits. �e concept of focus cue generation is based on superbeam-shaping condition, as will be described in Section 6. For the
3.3 Dynamic
Eye-boxmulti-view
displays or light �eld displays. However, in Retinal 3D,
single projector
the focalof lensesone
L1, L2,
In prototype,this section,
welengths
introduceofHIC
the key concepts of th32
For realizing an AR HMD, a VRD has powerful advantages
as dewhich is localized light©�eldwith pupil-tracking
2017scanning
2015
Takafumi KOIKE
ACM Transactions on Graphics, Vol. 36, No. 6, Article 190. Publication date: November 2017.
scribed in Section 2.4. However, the size of the limited eye motion
�eld displays, focus cues are provided by sampling the li

ACM Transactions on Graphics, Vol. 36, No. 6, Article 190. Publication date: November 2017.

4. ライトフィールドディスプレイ
関連企業

2017 Takafumi KOIKE© 2015

スタートアップ企業• 米国を中心にライトフィールドディスプレ
イのスタートアップ企業が続々現れている• 注目を浴びている企業を中心に紹介する．
以前の3Dディスプレイブームと似た雰囲気があるが…

34

2017 Takafumi KOIKE© 2015

Avegant Corp.

35

2017 Takafumi KOIKE© 2015

Avegant Corp.
以下の3点を売りにして，GlyphとLight
Fieldの処理技術を用いてMR向けプラット
フォームを開発．

• Virtual Image Clarity• Simple Software Development• Easy Hardware Integration36

2017 Takafumi KOIKE© 2015

Avegant Corp.

http://www.kguttag.com/2017/03/09/avegant-light-ﬁeld-display-magic-leap/ より引用
37

2017 Takafumi KOIKE© 2015Light Field Lab.

• ホログラフィックディスプレイを開発中• 2018年に開発者向けサンプルを出荷予定

382017 Takafumi KOIKE© 2015

Leia 3Dhttps://www.leia3d.com• HP Lab.出身のDavid Fattalによって設立されたベンチャー企業• 多方向バックライトが特徴

39

2017 Takafumi KOIKE© 2015

多方向バックライト

D. Fattal, Z. Peng, T. Tran, S. Vo, M. Fiorentino, J. Brug, and R. G. Beausolei. “A Multi-directional Backlight for a Wide-angle,
Glasses-free Three-dimensional Display,” Nature, 495, pp. 348-351, 2013. より引用
40
2017 Takafumi KOIKE© 2015

Hydrogen One Phone• 映画用カメラメーカのRED社によるスマー
トフォン• 2017/9/7にLeia社と提携• 3Dスマートフォンを2018年夏に発売予定• 2Dはフル解像度• 3Dは4視点
http://downloads.red.com/hydrogen.pdf
41© 2017,2018 Takafumi KOIKE© 2015Lumi• MITにいたMatthew
Hirschらが立ち上げた• Gordon Wetzsteinも関わっているhttp://www.lumiidisplay.com
422017 Takafumi KOIKE© 2015

まとめ• ライトフィールドディスプレイの復習• 最新の研究を見ると，ディスプレイ系学会だ
けでなく，コンピュータサイエンス系の学会での発表が増えている• 原理的に新規性が高い提案は少ないが，より
複雑(光学系+センシング+ソフトウェア)なシ
ステムの実装例が増えている• アプリケーションの模索も盛んになっている
43© 2016 Takafumi KOIKE© 2015,