Light Field Blender: Designing Optics and Rendering Methods
for See-Through and Aerial Near-Eye Display
Kazuki Otao

Yuta Itoh

Hiroyuki Osone

University of Tsukuba
Pixie Dust Technologies, Inc.
kazuki.otao@pixiedusttech.com

University of Tsukuba
Pixie Dust Technologies, Inc.
s1711428@s.tsukuba.ac.jp

University of Tsukuba
Pixie Dust Technologies, Inc.
soneo.tsukuba@gmail.com

Kazuki Takazawa

Shunnosuke Kataoka

Yoichi Ochiai

University of Tsukuba
Pixie Dust Technologies, Inc.
kazuki.takazawa@pixiedusttech.com

National Institute of Technology,
Tokuyama College
i12kataoaka@tokuyama.ac.jp

University of Tsukuba
Pixie Dust Technologies, Inc.
wizard@slis.tsukuba.ac.jp
Aerial LightField Image(d)LCD Case
LCD
deθ

Frame ofLens Array
Lens Array

See-through background object

Frame of
TMD Plate
TMD PlateFigure 1: (a) Concept of enabling a wide field of view, near-eye, see-through display using Transmissive Mirror Device. (b) Use
case for our HMD. The HMD consists of a LCD, a micro-lens array, and a TMD with micro-mirror array. (c) The image obtainedwith the aerial light field image and the background object image. (d) Structure of our proposed HMD.

ABSTRACTCCS CONCEPTS

In this study, we propose a novel head-mounted display (HMD) design for near-eye light field display which achieves a see-throughand wide field of view for augmented reality. In the past years,
many optical elements such as half-mirror, beamsplitter, and waveguide were employed for an optical see-through display. We use atransmissive mirror device (TMD) instead of conventional opticalelements. A TMD consists of numerous micro-mirrors and is usually used for real imaging system in the mid-air. We introduce a
new method for the TMD plate in order to extend the previous
near-eye display to the see-through display. We achieve a wide
field of view and comfortable viewing by creating a point lightsource with a micro-lens array. Our configuration is very simple
and consists of a LCD for the image source, a micro-lens array toprovide the light field, and a TMD plate to provide the aerial image
in front of the eye. We construct a prototype see-through display
including a fabricated HMD. We verify the design of our prototype using simulations and experiments, and further discuss thechallenges in building a novel near-eye, see-through display.

• Hardware → Displays and imagers;

SA ’17 Technical Briefs, November 27–30, 2017, Bangkok, Thailand© 2017 Association for Computing Machinery.
This is the author’s version of the work. It is posted here for your personal use. Not
for redistribution. The definitive Version of Record was published in SA ’17 Technical
Briefs: SIGGRAPH Asia 2017 Technical Briefs, November 27–30, 2017, Bangkok, Thailand,
https://doi.org/10.1145/3145749.3149425.

KEYWORDS
Augmented Reality, Near-Eye Display, Light Field, Optical See-ThroughDisplay
ACM Reference Format:
Kazuki Otao, Yuta Itoh, Hiroyuki Osone, Kazuki Takazawa, Shunnosuke
Kataoka, and Yoichi Ochiai. 2017. Light Field Blender: Designing Optics
and Rendering Methods for See-Through and Aerial Near-Eye Display. In
SA ’17 Technical Briefs: SIGGRAPH Asia 2017 Technical Briefs, November 27–
30, 2017, Bangkok, Thailand. ACM, New York, NY, USA, 4 pages. https://doi.
org/10.1145/3145749.3149425

1 INTRODUCTION
Near-eye, see-through displays for augmented and mixed reality
have been recently proposed for various uses to the general public.
In the commercial field, Microsoft Hololens and Meta are available
to end users. In such a use case, an immersive experience is desired. However, there is still room for research on optical elements
and presentation methods for visual information because the fieldof view is still narrow. In addition to a high immersive experience,
we also have to consider distorting scenery and decreasing scenerybrightness. To solve such problem, many optical elements such astransmissive LCD, half-mirrors, freeform optics, waveguides, andSA ’17 Technical Briefs, November 27–30, 2017, Bangkok, Thailand
holographic optical elements have been used for an optical seethrough display. These optical elements have limitations of viewing angle, distortion and brightness, and hence, perfect optical elements are not realized. Thus, it is necessary to explore near-eye
optical elements further.
Several papers have reported that the light field display has considerable advantages for a near-eye display [Lanman and Luebke
2013]. For example, the light field display reproduces a correct rayof light and solves the accommodation convergence conflict. A seethrough light field display has also been proposed [Maimone andFuchs 2013] by other papers.
In this study, we introduce a novel method to present the lightfield directly in front of the eye by using a transmissive mirror
device (TMD) plate that consists of micro-mirror array.
Our primary technical contributions are:
• We introduce a TMD plate, which is usually used for aerial
imaging for the near-eye, see-through display.
• We analyze the optical light ray in our configuration, consisting of a micro-lens array and a TMD plate, by using computer simulation.
• We present the possibility of usage of the aerial light field
image for near-eye display through experiments.
We explored the combination of a near-eye light field display
and a TMD plate. Although a TMD plate adapted for an eyepiece
was proposed [Ochiai et al. 2017], this is one of the first approaches
to use a TMD plate for the near-eye light field display.

2

RELATED WORK

Our proposed method corresponds to the near-eye light field display and the aerial imaging using a TMD plate. We introduce someresearch about the near-eye display that provides the light field image. We also introduce the aerial imaging system and interactions
using TMD plate.

2.1

Light Field for Near-Eye Display

The integral imaging and the light field technology used for neareye display has an advantange that it reproduces the correct light
ray and solves the accommodation convergence conflict. Several
light field displays for near-eye display were proposed [Lanman
and Luebke 2013] [Huang et al. 2015]. Some researchers also aimed
to obtain a see-through capability. Hua and Javidi presented a research that makes the light field with micro-lens array, free fromoptical elements [Hua and Javidi 2014]. It achieved high brightness
like a real world scene, however, it had a very narrow field of view.
Maimone and Fuchs stacked Transmissive LCD panel [Maimone
and Fuchs 2013]. It could reproduce the light field, but had a low
spatial resolution. Pinlight Displays [Maimone et al. 2014] madepoint light source and modulated it. Because of the modulationmask, brightness of the real world scenery through the transmission glasses is decreased. Holographic for near-eye [Maimone et al.
2017] has a high potential that provides a correct light ray and highresolution. Furthermore it could solve the accommodation convergence conflict. However, it requires a high computational performance. Recent varifocus displays were easily realized by deformation optics or movable optics without computational cost [Dunnet al. 2017] [Akşit et al. 2017].

K. Otao et al.

Image Source

Side View(Corner Reﬂector)

Top View(Transmissive)

TMD Plate

TransﬀeredReal Image

Figure 2: (a) Structure of Micro Dihedral Corner Reflector
Array employed as a TMD plate. (b) The TMD plate transfers
the real image to the plane symmetric position.

2.2

Aerial Imaging System using TMD plate

TMD consists of numerous micro-mirrors and several structures
such as Micro Dihedral Corner Reflector Array (DCRA) [Maekawa
et al. 2006] [Yamane et al. 2015]. TMD plate is usually used for anaerial imaging system and an aerial interaction. HaptoMime [Monnai et al. 2014] present an aerial interaction system that allows
users to touch a floating virtual screen with hands-free tactile feedback. HaptoClone [Makino et al. 2016] propose an interactive system that mutually copies adjacent 3D environments optically andphysically. However, there is still a scope for research on the TMD
plate for near-eye display. Ochiai et al introduce this TMD plate
for the near-eye display [Ochiai et al. 2017]. In our method, we
achieved optical see-through light field display without the scenerydistortion using a TMD plate. Furthermore by using a micro-lensarray instead of the eyepiece lens, a natural image without distortion is obtained.

3 PRINCIPLE
Before discussing the whole aspect of our method, we describe thebehavior of a TMD plate and the near-eye light field display witha micro-lens array. The TMD plate transfers the real image to theplane symmetric position so that we can place the light field directly in the aerial in front of the eye.
Transmissive Mirror Device plate. In this research, we employ
Micro DCRA as the TMD plate. DCRA consists of numerous micromirrors placed perpendicular to the surface of the substrate. The
structure of DCRA is shown in Figure 2 (a). The light rays incident on the TMD plate are reflected by a micro-mirror and pass tothe opposite side. Although the principle of operation is based onreflection by mirrors, the device is also transmissive and deflects
light. It creates a floating image shown in Figure 2 (b).
Near-eye Light Field Display with Micro-lens Array. Near-eye lightfield display with a micro-lens array was formulated by [Lanman
and Luebke 2013]. To create the light field image, we define planeof focus in a virtual scene. The virtual camera represents each lensas placed in a virtual scene. The virtual scene is rendered with an
off-axis projection and the process will repeat for the times equal
to the number of lenses. Each rendered element is displayed on the
LCD and each lens magnifies the LCD elements.

Light Field Blender

SA ’17 Technical Briefs, November 27–30, 2017, Bangkok, Thailand
Image Source

Lens Array

Lens Array

Eye

Eye

Image Source(d)

TMD plate

TMD plate

Eye

Figure 3: (a), (b), (c) Optical ray tracing in computer simulation. (a) LCD and micro-lens array. (b) LCD and TMD plate. (c) LCD,
micro-lens array, and TMD plate. (d) Displayed image on the LCD.
Our Proposed Method. We transfer the light field image generated by the micro-lens array in front of the eye. Figure 2 (c) shows
our proposed principle. The light source emitted from the LCD isconverted to the light field image by the micro-lens array. The lightfield image generated by the micro-lens array can be seen near theeyes, however, it is not optical see-through. The TMD plate transfers the light field image to the plane symmetric position. Thus, we
can observe aerial light field image with the real scenery.

4

SIMULATIONThe combination of the micro-lens array and the TMD plate makes
complex light rays. In order to reveal the behavior of complex lightrays, we simulate optical ray tracing. Optical ray tracing in computer simulation was performed using Zemax OpticStudio. In Figure 3, we present how the light ray behaves in the near-eye opticalsystem. In the LCD configuration, the image is blurred out of focus,
and hence, it cannot provide a near-eye image. In Figure 3 (a), LCD
and micro-lens array allow to use the image for near-eye, however,
it is not optical see-through. Figures 3 (b) and 3 (c) show optical
ray tracing with TMD plate. TMD structure with Micro DCRA was
reproduced using CAD. Red, green, and yellow-green lines reproduce point light sources which are emitted from LCD. Blue lines
reproduce an optical ray which comes from the real world scenery.
LCD and TMD plate configuration provide the aerial image andcapability of optical see-through. However, it is not suitable fornear-eye image. The configuration used in our method consists ofa LCD, a micro-lens array and a TMD plate that enable optical seethrough with wide viewing angle.

5 IMPLEMENTATION
5.1 Prototype Head-Mounted Display
We build our prototype with a fabricated HMD as shown in Figure 1 (b). Our configuration consists of a: LCD for the image source,a micro-lens array to provide the light field, a TMD plate to provide the aerial image in front of the eye. We employ Sony Xperia
XZ Premium Dual G8142 as LCD. The display size is 121cm × 68cm,
the pixel resolution is 3840 × 2160pixel, and the ppi is 801. However, as we use only a quarter of the LCD, we can convert it to asmaller one. In the micro-lens array, the lens pitch is 1.98mm andfocal length is 10.0mm. The distance from LCD to micro-lens arrayis de = 0.5mm. We select Micro Dihedral Corner Reflector Arrayas the TMD plate. The mirrors are alternately arranged at an interval of 0.3mm. The inclination of the installed TMD plate is θ = 45◦ .
The frame of HMD was 3D printed by Makerbot Replicator+ 1 . The
weight of our HMD is 450д.

5.2

Rendering Software

In order to create the image display on the LCD, we place a virtual
camera representing each lens of the micro-lens array in the virtual
scene. The virtual camera uses an off-axis projection frustum. We
implement this process using the software by Unity. An example ofthe image displayed on the LCD is shown in Figure 3 (d). A photoof a prototype display using a camera is shown in Figure 1 (c).

6 EXPERIMENTS
We evaluated a retinal blur of the light field image. We placed aStanford bunny behind our HMD and rendered an Utah teapot
on our HMD. The distance of the Stanford bunny from the Utahteapot was 20cm. We changed the focal length of the camera from50cm to 70cm. Figure 4 (a) shows a photo focused on an aerial lightfield image of an Utah teapot. Figure 4 (b) shows a photo focusedon a background image of a Stanford bunny. The retinal blur wasachieved by an aerial light field image.
We also evaluated the extent of the distortion of the light field
image transferred in the aerial by the TMD plate distances. Figure 5 shows the results when the transferred light field image andthe camera distances were 25cm, 30cm, 35cm, 40cm, respectively.
The results show that the light field image in the aerial is not distorted by distance. Then, we moved the camera back and forth, andcaptured aerial images near the focal length (Figure 6). Focal lengthof a camera is f = 100mm, ISO sensitivity was 1000.

7 DISCUSSION
In this section, we describe the advantages and limitations of our
HMD. Our limitations depend on the TMD plate and can be solved
by replacing other TMD structures.
Aerial Image without Distortion. Our HMD with micro-lens array and TMD plate does not cause any distortion. Half-mirror seethrough display is known to cause a distorted image from the scenery.
It is also known that the eyepiece lens distorts a light source, andhence, an optical solution or a software solution is required. OurSA ’17 Technical Briefs, November 27–30, 2017, Bangkok, Thailand

(a)

(b)

K. Otao et al.
To cope this problem, we have to consider to cut unnecessary lightpath.

8 CONCLUSION

Figure 4: (a) A photo focused on the aerial light field image.
(b) A photo focused on the background object.
A：25cm

B：30cmC：35cmREFERENCES

D：40cmFigure 5: The aerial image distortion by distance from the
TMD plate.

(a)

(b)

(c)

(d)

+1cm
+1cm

f

-1cm

-2cm

d

CameraIn this paper, we introduce a novel HMD design for a near-eye, seethrough display for augmented reality. We use a TMD plate insteadof conventional optical elements. We were able to provide the lightfield image directly in front of the eye by combining the light field
display and the TMD plate. Our proposed HMD provides correct
light ray and reproduces retinal blur. Additionally, our proposed
HMD does not cause the scenery and the aerial image distortion.
We believe that this is an advantage over other optical HMDs.

TMD
Lens
Array
LCD

Figure 6: Optical setup. (a) d = f +1cm. (b) d = f . (c) d = f −1cm.
(d) d = f − 2cm.

HMD employs a micro-lens array and a TMD plate so that a correct
image is generated without any optical solution.
See-through Capability. As shown by the simulation result of
the Figure 3 (d), a TMD plate with Micro DCRA reduces brightness
from the scenery. There is a possibility that this issue can be solved
by adopting a different structure of the TMD plate.
An Undesired Image by double reflection. The TMD plate causes
double reflection images to appear diagonally (Figure 1 (c)). Thisis undesirable because it shows an unintended image to the user.

Kaan Akşit, Ward Lopes, Jonghyun Kim, Josef Spjut, Anjul Patney, Peter Shirley, David
Luebke, Steven A. Cholewiak, Pratul Srinivasan, Ren Ng, Martin S. Banks, andGordon D. Love. 2017. Varifocal Virtuality: A Novel Optical Layout for Near-eye
Display. In ACM SIGGRAPH 2017 Emerging Technologies (SIGGRAPH ’17). ACM,
New York, NY, USA, Article 25, 2 pages. https://doi.org/10.1145/3084822.3084829
David Dunn, Cary Tippets, Kent Torell, Henry Fuchs, Petr Kellnhofer, Karol
Myszkowski, Piotr Didyk, Kaan Akşit, and David Luebke. 2017. Membrane AR:
Varifocal, Wide Field of View Augmented Reality Display from Deformable Membranes. In ACM SIGGRAPH 2017 Emerging Technologies (SIGGRAPH ’17). ACM,
New York, NY, USA, Article 15, 2 pages. https://doi.org/10.1145/3084822.3084846
Hong Hua and Bahram Javidi. 2014. A 3D integral imaging optical see-through headmounted display. Opt. Express 22, 11 (Jun 2014), 13484–13491. https://doi.org/10.
1364/OE.22.013484
Fu-Chung Huang, Kevin Chen, and Gordon Wetzstein. 2015. The Light Field Stereoscope: Immersive Computer Graphics via Factored Near-eye Light Field Displays with Focus Cues. ACM Trans. Graph. 34, 4, Article 60 (July 2015), 12 pages.
https://doi.org/10.1145/2766922
Douglas Lanman and David Luebke. 2013. Near-eye Light Field Displays. In ACM
SIGGRAPH 2013 Emerging Technologies (SIGGRAPH ’13). ACM, New York, NY, USA,
Article 11, 1 pages. https://doi.org/10.1145/2503368.2503379
Satoshi Maekawa, Kouichi Nitta, and Osamu Matoba. 2006. Transmissive opticalimaging device with micromirror array. (2006), 63920E-63920E-8 pages. https://doi.org/10.1117/12.690574
Andrew Maimone and Henry Fuchs. 2013. Computational augmented reality eyeglasses.. In ISMAR. IEEE Computer Society, 29–38. http://dblp.uni-trier.de/db/
conf/ismar/ismar2013.html#MaimoneF13
Andrew Maimone, Andreas Georgiou, and Joel Kollin. 2017. Holographic Near-Eye
Displays for Virtual and Augmented Reality. ACM Transactions on Graphics 36(July 2017), 85:1–85:16. https://www.microsoft.com/en-us/research/publication/
holographic-near-eye-displays-virtual-augmented-reality/
Andrew Maimone, Douglas Lanman, Kishore Rathinavel, Kurtis Keller, David Luebke,and Henry Fuchs. 2014. Pinlight Displays: Wide Field of View Augmented Reality
Eyeglasses Using Defocused Point Light Sources. ACM Trans. Graph. 33, 4, Article
89 (July 2014), 11 pages. https://doi.org/10.1145/2601097.2601141
Yasutoshi Makino, Yoshikazu Furuyama, Seki Inoue, and Hiroyuki Shinoda. 2016. HaptoClone (Haptic-Optical Clone) for Mutual Tele-Environment by Real-time 3D Image Transfer with Midair Force Feedback. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). ACM, New York, NY, USA,
1980–1990. https://doi.org/10.1145/2858036.2858481
Yasuaki Monnai, Keisuke Hasegawa, Masahiro Fujiwara, Kazuma Yoshino, Seki Inoue, and Hiroyuki Shinoda. 2014. HaptoMime: Mid-air Haptic Interaction with a
Floating Virtual Screen. In Proceedings of the 27th Annual ACM Symposium on User
Interface Software and Technology (UIST ’14). ACM, New York, NY, USA, 663–667.
https://doi.org/10.1145/2642918.2647407
Y. Ochiai, K. Otao, and H. Osone. 2017. Air Mounted Eyepiece: Design Methods for
Aerial Optical Functions of Near-Eye and See-Through Display using Transmissive
Mirror Device. ArXiv e-prints (Oct. 2017). arXiv:cs.HC/1710.03889
T. Yamane, S. Maekawa, Y. Utsumi, I. Okada, and A. Yamaguchi. 2015. Fabrication andevaluation of Dihedral Corner Reflector Array for floating image manufactured
by synchrotron radiation. In 2015 International Conference on Electronics Packaging and iMAPS All Asia Conference (ICEP-IAAC). 436–439. https://doi.org/10.1109/
ICEP-IAAC.2015.7111052