3D4/VHF6 - 1

HaptoMIRAGE: An Active-shuttered Real Image
Auto-stereoscopic Display
Yuta Ueda*, Hideaki Nii**, Kouta Minamizawa*, Susumu Tachi****Keio University, 1-1-4 Hiyoshi, Kohoku-ku, Yokohama, Kanagawa, 223-8526, Japan**IIJ Innovation Institute Inc., 2-10-2 Fujimi, Chiyoda-ku, Tokyo, 102-0071, Japan**University of Tokyo, 1-2-7, Hongo, Bunkyo-ku, Tokyo, 113-8526, Japan
Keywords: Active shutter, Aerial 3D Image, Auto-stereoscopic display

ABSTRACT
HaptoMIRAGE is an auto-stereoscopic display that
does not require the viewer to wear devices like 3D
glasses. HaptoMIRAGE can project 3D images in mid-air.
Auto-stereoscopic images are projected in mid-air, and areal object is placed near the image position, making it
easy for the user to gaze at the image. Therefore, it isconvenient to produce content such that real objects and3D virtual objects are combined. Furthermore, multiple
users can share the 3D view from their respective
viewpoints.

1. INTRODUCTION
Our purpose is to implement a platform for storytelling,entertainment,
andcreative
collaboration
bysuperimposing auto-stereoscopic 3D images on real
objects such that they can be observed by multiple users.
If the 3D content is naturally manipulated by the user, it isnecessary to meet the point that users can handle 3D
image with real objects. Users should be able to close the3D image and reach out with their hands to the 3D image.
Therefore, the 3D image should be projected in mid-air.
Furthermore, when we handle real objects, we usuallyshare the objects with other people. Likewise, the 3D
image should be visible to multiple users from theirrespective viewing positions because users usually view
the objects from a free viewing angle.
To experience real objects and virtual objects combinedin real space, it is imperative develop a 3D display
technology that can project a 3D image in midair, whereinmultiple users can view the 3D image from theirrespective viewing positions. In addition, it is also
important to ensure that the users can experience theprojected 3D image like a real object—by using theirhands. The proposed 3D display enables users to reachout with their hands to the 3D image effortlessly, and puts
real objects in close proximity to the 3D image.
2. RELATED WORKS
To superimpose 3D images on physical objects in
mid-air, we require a 3D display technology that canproject 3D images in mid-air. To address this requirement,a number of 3D display systems have been developed.
For example, HoloAd [1], developed by InnoVision Labs,
can superimpose 2D images on real objects, whereinmultiple users can view and share the 2D computer
graphic (CG) image and the real object. This system has
been used at exhibitions and for some advertisingstratagems. MRsionCase by Naemura et al. [2] is anadvanced extension of HoloAd that enables viewing of
the 2D image from approximately 360 degrees. Thesedisplays are visible to multiple users as 2D images
spread over a wide area. However, this technology islimited in that users cannot reach and touch the CG
image and the object with their hands. To address thisissue, Naemura [3] proposed the Mid-air Augmented
Reality Interaction with Objects (MARIO) that canproject the CG image in midair, and users can touch andexperience the real and CG objects.
The displays of the above approaches can project CG
images in midair. However, if 3D actions, such as airdrawing, are performed, users must be able to view theimages in 3D. To address this requirement, light field
display technology [4] is employed to enable users toobserve the 3D image in a 360-degree field byprojecting the light irradiated from the projector to ahigh-speed rotating mirror. This projector is arranged tocorrespond with the angle at which the mirror ispositioned. An Auto-stereoscopic Projector Array
Optimized for 3D Facial Display, developed by Naganoet al. [5], enables multiple users to view the 3D CGobjects. This system projects a 3D image through thearrangement of several projectors. Although the systemenables viewing of the 3D image, the 3D image
projection point is at the point of the rotating mirror or
screen; therefore, users cannot touch the 3D image orplace physical objects in the scene. FLOATS [6], on theother hand, uses active shuttered auto-stereoscopic
methods to enable users to reach their hand onto the 3DCG image. The active shuttered auto-stereoscopy
method, which does not decrease resolution, provides
the auto-stereoscopic image using a slit that aligns tothe user’s head. In this manner, the active shutter
method provides only two auto-stereoscopic images,
which correspond to each eye. In addition to enablingusers to touch the 3D image, this technology does not
require the preparation of the image for every viewpoint
to cover 360 degrees for a panoramic view. This system
only requires the preparation of 2 images corresponding
to users active head position. Active shuttered method
enables the use of minimal image data, such as twoimages for each corresponding eye thus addressing thelimitation of tracking the user’s head position. RePro3D[7] can project a multi-view 3D image on a diorama inreal space by applying retro-refractive projection
technology. The retro-refractive projection enables theuser to view the CG image by aligning the image to theuser’s eye position. Therefore, the user can view thecorresponding version of the image from each eye.
A 3D display for multiple users, fVisiOn [8] can project

ISSN-L 1883-2490/22/0796 © 2015 ITE and SID

IDW ’15796

a 3D image in midair, and the 3D image can be viewed
from a 130-degree wide-angle view. This system
precisely accommodates the multiple user experience;nevertheless, it is limited in terms of projection space.
Furthermore, it can only project the image on a table.
Therefore, interaction with this system is restricted to the3D image movement area and the positional relationshipbetween real objects and the 3D drawing.
3. ARIA METHODSTo enable the superimposition of a 3D image on real
objects and to enable multiple users to manipulate the 3D
image with real objects, we employed our Fuwa-Vision 3D
display [9], which can project the 3D image in midair using
the active-shuttered real image auto-stereoscopy (ARIA)
method, which is similar to FLOATS, that provides
auto-stereoscopic images to track user’s eye position.
This method produces an auto-stereoscopic image using
an active shutter and transparent Liquid crystal display(LCD). It projects a real image in mid-air using a Fresnellens.
Figure 1 illustrates the ARIA system overview. Each
ARIA module consists of an LCD, a transparent LCD, anda Fresnel lens. The LCD provides the auto-stereoscopicimage; the transparent LCD is used for the active shutter,and the Fresnel lens is used to project the real image inmidair. These components work together using the ARIA
method and 3D image auto-stereoscopy in mid-air.
Although the displays can produce a 3D image in midair,
the viewing angle is narrow—approximately 40 degrees;
therefore, interaction by multiple users is not feasible.
Figure 2 illustrates the ARIA method principles. Whenthe position of the convex lens is set at x = 0, the positionof the active shutter, which is used by the transparent LCD
for dynamic changing of the slit width and position, is set
at x = -S3. The position of the LCD is set at x = -S1. Theeye position of x = S4 is the conjugate point from theactive shutter across the lens. The slit changes so that theauto-stereoscopic image is projected to align with eacheye; accordingly, it can provide the auto-stereoscopic
image by time division. The 3D image projects at x = S2;
S2 is determined by S1 and the focal point of the lens.
To project the 3D image at a position that is preciselyaligned to the user’s eye position, the slit width andposition must be determined to prevent vignetting andprojecting of an auto-stereoscopic image that does notcorrespond to the user’s eye position. As shown in Figure
2, the conjugate point of the eye position is (u, v), theposition of the 3D object’s upside and downside are (-S1,i) and (S3, sy), respectively, and the position of the slit
node end point is (S3, sy).
Therefore, sy, as defined above, is defined by theformula as follows.
    (1)           (2)    

Eq. (2) indicates that the auto-stereoscopic image is notprojected to correspond with the viewer’s eye position.
However, Eq. (1) represents the case in which the eye

797

IDW ’15

position is set on an optical axis. Therefore, if the eye
position moves from the optical axis, Eq. (1) must
likewise involve a derivative.
4. SYSTEM STRUCTURE
The detailed overview of the HaptoMIRAGE 3D
display system is shown in Figure 3. The system
consists of three ARIA units. Each unit is comprised of aDell P2213 (22 inch, 1680x1050 pixels) for the LCD, aSamsung LTI220MT02 (22 inch, 1680x1050 pixels) for
the transparent LCD, and two combined NTKJ
CF400-0.3 Fresnel lenses (focal distance of 400 mm).
As shown in Figure 2, the LCD is set at S1 = 400 mm,
and the transparent LCD is set at S3 = 220 mm. The
projecting point determines S2 = 400 mm, and the userbase position determines S4 = 2000 mm.
To track the head position, a Kinect v2 motion tracking
camera system is used. The active shutter follows theuser’s head and provides 60 frames per second images
to the spot of user’s eye. In this system, one ARIA unit
uses one PC; therefore, this system employs three ARIA
units that render the virtual image in mid-air. Thus,
HaptoMIRAGE 3D display makes it possible to view the3D image from a wide-view angle using three ARIA
modules.
5. USER INTERACTION
HaptoMIRAGE enables the users to view 3D images
and interact with real objects suspended in mid-air.
These combined capabilities make possible the novel
virtual reality field interactions that were formerly notpossible. These interactions involve superimposing
one’s own operating onto physical objects andinteracting with one’s own operating with dynamicmotion, such as changing the view point with multiple
viewing angles, rotating the base to view different
aspects. Moreover, 3D CG effects, such as rain, fire,and snow etc., can be superimposed on real objects—afeature that bridges real and virtual worlds. Theseinteractions
additionallyinclude
themid-airsuperimposition of dynamic 3D CG content, such as
character animations, collaborative drawing amongmultiple users, real-time drawing and discussion. These
interactions are detailed in the following sections.
5.1 Interact with physical objects
As described earlier, HaptoMIRAGE can project a 3D
image on physical objects. Therefore, we can use thissystem for exhibiting some artistic objects orcommodities, such as toys, using a 3D image.
Subsequently, we can also apply active effects, such asfire, rain, and snow to real objects in a virtually. Figure 4,
for example, shows the 3D image of a car superimposedon a real stage-like platform. The user can rotate the 3D
image by using the rotating stage and can place objects
on the real stage.
As shown in the Figure 4 example, when the user
rotates the stage using a motor to control the angle ofrotation, the 3D image rotates correspondingly bytracking the stage rotation angle. This correlation
enables the user to draw more complicated sketches inmid-air from different angles. Furthermore, users cansuperimpose their 3D sketch on real or CG objects. This
capability makes it possible to expand the compatibility ofcreative exhibits/displays from the restrictive 2D format to3D format with real objects, such as art exhibits, toys, and
3D CG.
5.2 Creating 3D objects in midair
Using the HaptoMIRAGE capability of superimposing a
3D image in mid-air, we implemented an air sketch
application, as shown in Figure 5. Users can draw theilluminated lines in mid-air by using a pen shaped device.
The device is equipped with a button that the user canpress while drawing the illuminated lines. Thus, creativeartwork, such as sketching, which is typically performedon the 2D field of an LCD or canvas, can be transformed
from a 2D field to a 3D field. Furthermore, multiple users
can collaboratively draw lines in the real environment andview it from various perspectives. Conversely, the user
can create a sketch on a real object. When a prototype
sketch is created with real objects, the user can then
combine it with a 3D drawing or sketch.
6. DISCUSSION
HaptoMIRAGE system enables the projection of a 3D
image in mid-air, and multiple users can interact with the3D image in midair using their hands from respective
viewing positions. HaptoMIRAGE has a lot of potential for
mixed reality environments such as those described inchapter 5. HaptoMIRAGE is advantageous in certain
aspects compared to other 3D displays systems.
Firstly, HaptoMIRAGE system enables users to view the3D image naturally. Multiple users can view the 3D image
from their respective positions, and they can see the 3D
image at various angles. HaptoMIRAGE system does not
require additional connecting equipment between theusers and 3D image. Therefore, the users can experiencea mixed reality environment without stress or interferencefrom any obstacles. Furthermore, it is does not require theusers to wear the any viewing glasses to see the 3Dimage. Users can see the 3D image just closing toHaptoMIRAGE system without wearing any devices.
Secondly, the HaptoMIRAGE system can be constructedeasily and is relatively cost effective compared with anyother 3D mixed reality display system. HaptoMIRAGE
system consists of three ARIA units, and the ARIA units
comprise an LCD, transparent LCD, and Fresnel lens.
There are also certain drawbacks to using theHaptoMIRAGE system. Firstly, the 3D image cannot be
projected in front of real objects. The 3D image isprojected using lenses; therefore, if we place real objects
in close proximity to the 3D image, users will see the 3D
image behind the real objects. HaptoMIRAGE system ismakes it plausible to interact with the 3D image using ourhands or real objects; however, it is found wanting whenplacing real object near the 3D image. It is possible toplace real objects in the proximity, but there are some
limitations to the scale and placement position of the realobjects. Secondly, HaptoMIRAGE system can project the3D image for viewing for up to 3 users. This is because
the ARIA units can project the 3D image for a single
viewer because of the refresh rate limitation. The current
system projects the auto-stereoscopic image to user’s
right and left eye to flip by 60Hz. The user’s eyes can view

the auto-stereoscopic image at 30Hz individually. It ispossible for ARIA units to project the 3D image for morethan two users, but whenever the number of users
increases, the refresh rate for user’s individual eye
decreases. For increasing number of users, it isimperative to use higher refresh rate display units(including LCD, and transparent LCD) instead of thecurrent LCDs used in this study.
7. CONCLUSION
In this study, we presented the use of anauto-stereoscopic display for seamless interaction with
mixed reality environments. The proposed system canproject a 3D image in mid-air with a 150-degree
wide-angle view based on our proposed ARIA
technology. Up to three users can observe the sameimage from different perspectives based on their relative
position with respect to the projection. The 3D imagecan be superimposed on a real object, which allows theuser to naturally interact with the mixed reality
environment. Therefore, multiple users can experience
coexisting Virtual and Real environment. Applications of
the system include superimposing 3D images on objects
in real environments like exhibitions and toys. In
addition, users can draw auto-stereoscopic 3D
illuminated lines in real environments. HaptoMIRAGE
enables the users to bring virtual objects into the realworld, while simultaneously interacting with multiple
users. These capabilities can be applied to 3Dmodeling, collaborative drawing, idea sharing, and many
more applications in the future.
REFERENCES[1] HoloAD, InnoVision Labs.
http://www.innovision.com.tw/old_web/hoload.html[2] C.H. Hsu, C.C. Chiang, K.L. Hua & W.H. Cheng, “A
mixed-reality showcase for multiple users fromunconstrained viewing angles,” SIGGRAPH Asia
2013 Emerging Technologies (p. 1), ACM (2013)[3] H. Kim, I. Takahashi, H. Yamamoto, T. Kai, S.
Maekawa & T. Naemura, “MARIO: Mid-Air
Augmented Reality Interaction with Objects,”
Advances in Computer Entertainment (pp. 560-563),
Springer International Publishing (2013).
[4] A. Jones, I. McDowal, H. Yamada, M. Bolas & P.
Debevec, “An interactive 360 light field display,”
ACM SIGGRAPH 2007 Emerging Technologies, (p.
13). ACM (2007).

[5] K. Nagano, A. Jones, J. Liu, J. Busch, X. Yu, M.
Bolas & P. Debevec, “An autostereoscopic projector
array optimized for 3D facial display,” ACM
SIGGRAPH 2013 Emerging Technologies (p. 3),
ACM (2013).
[6] H. Kakeya, “P65: FLOATS V: RealImageBased
Autostereoscopic Display with TFTLC Filter,” SID
Symposium Digest of Technical Papers (Vol. 35, No.
1, pp. 490-493), Blackwell Publishing Ltd., (2004).
[7] T. Yoshida, K. Shimizu, T. Kurogi, S. Kamuro, K.
Minamizawa, H. Nii & S. Tachi, “RePro3D:
full-parallax 3D display with haptic feedback using
retro-reflective projection technology,” VR Innovation

IDW ’15798

(ISVRI), 2011 IEEE International Symposium on (pp.
49-54), IEEE (2011).

[8] S. Yoshida, “fVisiOn: Glasses-free tabletop 3-D
display—its design concept and prototype,” Digital
Holography and Three-Dimensional Imaging (p.
DTuA1), Optical Society of America, (2011).

[9] H. Nii, K. Zhu, H. Yoshikawa, N.L. Htat, R. Aigner & R.
Nakatsu,“Fuwa-Vision:anauto-stereoscopic
floating-image display,” SIGGRAPH Asia 2012
Emerging Technologies (p. 13), ACM (2012).

Fig3. System construction of HaptoMIRAGE
systemHaptoMIRAGE system is operated using one PC,
and is connected to three ARIA units.

Fig. 1 HaptoMIRAGE system overview
HaptoMIRAGE consist of three units: LCD, Active
Shutter, and Fresnel lens.

Fig5. Interaction with physical objects
Rotating the 3D image by rotating the physicalstage.

Fig6. Creating 3D image in midair
Real time illuminated lines drawn in mid-air.

Fig. 2 The principles of Active Shuttered Real
Image Auto-stereoscopy methods.
First image shows how to decide the slit position
correspond to user’s eye position.

799

IDW ’15