RePro3D: Full-parallax 3D Display with Haptic Feedback
using Retro-reflective Projection Technology
Takumi Yoshida∗

Keitaro Shimizu†

Tadatoshi Kurogi‡

The University of Tokyo

Keio University

Keio University

Sho Kamuro§Kouta Minamizawa¶

The University of Tokyo

Keio University

Hideaki NiiSusumu Tachi∗∗National University of Singapore

Keio University

A BSTRACTReal Object

Floating 3D Image
Haptic Device

We propose a novel full-parallax three-dimensional (3D) display
system-RePro3D-that is suitable for interactive 3D applications
with haptic feedback.Our approach is based on the retro-reflective
projection technology in which several images projected from a
projector array are displayed on a retro-reflective screen. Whenviewers view the screen through a half mirror, they see, without
the aid of glasses, a 3D image superimposed in real space. RePro3D has a sensor function that recognizes user input; therefore,
it can support some interactive features such as manipulation of 3D
objects. In addition, a wearable haptic device, which is a part ofour system, provides the user with a sensation of having touched
the 3D image. In this paper, we describe the optical system of thehigh-density projector array used in RePro3D. Then, we describe
the development of a prototype of RePro3D. The prototype is used
to demonstrate that our system displays full-parallax images superimposed in real space from 42 different viewpoints. The proposedsystem enables a user to physically interact with the 3D image withhaptic feedback.
Index Terms: H.5.1 [Information Interfaces and Presentation]:
Multimedia Information Systems—Artificial, augmented, and virtual realities
1

I NTRODUCTION

When virtual objects are projected in three dimensions, viewers
might get the feeling that the objects actually exist; these objects
can also be manipulated such that they behave as though they arebeing physically touched. The scenario described in the preceding
sentence has been depicted in science-fiction movies. It is expectedthat the use of stereoscopic displays in conjunction with user interface technologies will facilitate the realization of the abovementioned scenario.
In recent years, people have been enjoying three-dimensional(3D) images not only in movie theaters or amusement parks but
also at their homes. Most existing stereoscopic displays are basedon the concept of binocular stereo. A binocular-stereo-based display cannot render an accurate image with motion parallax andcannot create images that would provide different perspectives ofthe same image from multiple points of view. Motion parallax is
considered to play a very important role in the manner in which humans perceive 3D shapes. In order to simultaneously display more
∗ e-mail: takumi@tachilab.org† e-mail: k-taro@kmd.keio.ac.jp‡ e-mail: media kokubaru@hotmail.co.jp§ e-mail: kamuro@tachilab.org¶ e-mail: kouta@tachilab.org

 e-mail: nii@tachilab.org∗∗ e-mail: tachi@tachilab.org

Figure 1: Concept of RePro3D. A floating 3D image is projected inreal space. Users can touch the image directly.

natural stereoscopic images to a large number of people, multiview
displays that provide different perspectives according to viewing
directions are required. In addition, for a user to be able to viewthe displayed object as a real object, the displayed image should besuperimposed not on a screen but in real space.
Considerable research has been conducted on the developmentof user interfaces for interacting with 3D images. Multitouch technologies and vision-based gesture recognition are used to obtain inputs generated by the user’s hand movements. Haptic displays produce a tactile sensation when the user touches virtual objects. The
abovementioned technologies enable the users to interact with a 3D
image by means of intuitive movements. However, a system that reproduces the sensation produced when directly touching an object
has not yet been developed. One of the reasons is that the positionof the user’s hand is in different position of the displayed object. In
order to provide some degree of haptic feedback to the user’s hand,a haptic device should either be incorporated into the display or attached to the user’s hand. Most stereoscopic display screens have a
two-dimensional (2D) flat surface; however, for haptic interaction,
users move their hands in 3D space. Thus, it is difficult to combinea stereoscopic display and a haptic interface.
Therefore, the purpose of our study is to achieve the following
objectives:
1. generate autostereoscopic 3D images
2. superimpose 3D images in real space
3. develop a direct-touch interface for interacting with 3D images
To this end, we propose a novel full-parallax 3D display system
RePro3D that is suitable for interactive 3D applications with haptic
feedback. Figure1 shows the concept of RePro3D. For example, itwill be possible for a CG character to appear from a real dollhouseand for a user to physically interact with the character with haptic
feedback.
RePro3D is based on the retro-reflective projection technology(RPT). RPT has been researched to facilitate its use in developing visual display methods in augmented reality applications. Inthe conventional RPT method, an image is projected onto a retroreflective screen by a projector to superimpose a virtual image inreal space. In the proposed method, we use a high-density projector
array in place of a single projector. A number of images from the
projector array are projected onto the retro-reflective screen. Our
method can generate vertical and horizontal motion parallax. Whena user looks at the screen through a half mirror, he or she, without
the use of glasses, can view a 3D image that has motion parallax.
We can choose from two types of projection methods for superimposing 3D images in real space. We have named the two methods
as the real object projection method and the floating image method.
In the real object projection method, we can choose a screen shapedepending on the application. Image correction according to thescreen shape is not required. Therefore, we can design a touchsensitive soft screen, a curved screen , or a screen with an automatically moving surface. In the floating image method, the user candirectly touch the 3D image by wearing a haptic device on his/herhand.
In this paper, first, we describe the principle of RePro3D. Then,
we describe the implementation of the high-resolution optical system and the prototype device. Finally, we present a few results related to 3D projection and discuss the effectiveness of the proposed
method.
2 R ELATED W ORKS
A number of studies on multiview 3D displays have been conductedin the past. Multiview 3D displays are generally based on either the
integral imaging method[2, 3, 4] or the parallax barrier method[5].
In the integral imaging method, a lightfield is reproduced by placing an array of microlenses in front of the image. In the parallax
barrier method, a barrier with a number of slits is placed in front of
the image such that a different pixel is seen from different viewing
angles. Although our approach is somewhat similar to the integral
imaging method, it differs from both the methods mentioned above.
In the integral imaging method, the resolution of an image from aviewpoint depends on the size of lenses. Therefore, it is difficult
to improve the image resolution. In the parallax barrier method, it
is difficult to generate vertical parallax. On the other hand, in our
approach, the resolution of the image from a viewpoint depends on
the projector resolution. Therefore, it is easy to improve the image
resolution. In addition, our method can generate vertical and horizontal parallax. Further, the existing methods display a 3D image
on a flat screen, and cannot superimpose the image in real space.
Our method can be used to project a floating 3D image in real space,and users can view the projected image without the use of glasses.
Some projection-type display methods that superimpose images
in real space by using projectors have been proposed[6, 7, 8]. The
devices based on these methods project images on nonplanar, realsurfaces that users can actually touch, thereby allowing the users tointeract with the projected objects. However, such projected images
are 2D in nature and multiview parallax images cannot be generated. In this study, we aim to create a projection-type full-parallax
3D display system.
3 M ETHOD
3.1 High-density Projector Array
Figure2 shows the basic principle of the proposed RPT-based 3D
display method. When images from a projector array are projected
onto a retro-reflector, there is a strong reflection of light in the direction of each projection lens. An identical number of viewpoints
are created on either side of the axis of symmetry of the half mirror.
When a large number of projectors are arranged in a matrix, a 3D
image can be seen from multiple viewpoints.
In order to realize smooth motion parallax, the density of projectors in the projector array must be sufficiently high. However, withcommercially available projectors, it is difficult to form a projector

Floating 3D Image
Projector Array

Retro-reﬂector

Half Mirror

Figure 2: Basic principle of 3D display consisting of a projector array
and a retro-reflector.

array that has projectors located very close to each other. One of
the reasons for this is that the distance between adjacent viewpoints
is limited by the size of each projector. In addition, the system scale
would increase, and because of the large number of video outputs,
system cost would also increase. Therefore, we have developed ahigh-density projector array by arranging a number of projectionlenses in a matrix on a high-luminance LCD. The distance between
viewpoints depends on the size of the projection lens. A single
video output is produced, and the cost involved is lower than thecost of using many individual projectors. Figure3 shows the optical system of the high-density projector array. The system consistsof a number of lenses, an LCD, a half mirror, and a retro-reflector
that acts as the screen. The lenses are located at an appropriate distance from the image area of the LCD so that the projected areas ofthe projection lenses overlap. Shield plates are placed between thelenses to prevent light from other viewpoints from entering a lens.
Figure3 shows the lateral view of the optical system; the overhead
view would be identical to the lateral view.
The luminance of the projected image depends on the LCD luminance, viewing angle, and retro-reflector performance. Even if
the luminance of the projected image under normal illumination isso low that the image is not visible, it can still be seen at the focal
points . The resolution of the image from each viewpoint depends
on the LCD resolution. The number of viewpoints is equal to thenumber of projection lenses. This relationship between the resolution and the number of viewpoints is contrary to that in the integral
photography method. RePro3D can generate vertical and horizontal
motion parallax; however, the user must be positioned at a particular distance in the front-back direction .
3.2 Optical Design
Hereafter, we call the optical system of the projector array shown in
Figure3 as ”the basic system.” The following expressions hold true
for the basic system.
1
1
1
+
=
(1)
a0 b0
f0
h0
b0
=
a0
l

(2)

d0
b0
=
a0 + b0l(3)where a0 is the distance between the LCD and the projection lenses.
b0 is the distance between the actual image and the projectionlenses. l is the length of each image shown on the LCD. h0 is theReal Object Projection Method

LCD

Light Focus Points
Projection
LensesRetro-reflectorb0d0

IR Camera

Projector Array

Half Mirror
Half Mirror

a0Retro-reﬂector

Floating Image Method

Figure 3: Optical system of the high-density projector array.
LCD

Fresnel Lensb2a2
IR Camera
Retro-reﬂector

d1

Projector Array

d0

a1
b1
b0

Half Mirror

a0Figure 4: High-resolution optical system consisting of Fresnel lens.
Figure 5: Two projection methods for superimposing 3D images inreal space and IR-camera-based input interface.

size of the actual image shown on the LCD. f0 is the focal lengthof the projection lenses. d0 is the distance between adjacent projection lenses . On the basis of these expressions, we can determine thepositions and specifications of the lenses to be used in the system .
The smoothness of motion parallax depends on the distance between adjacent projection lenses, d0 . In order to increase the number of viewpoints, d0 should be minimized. However, if d0 is too
small, crosstalk will occur. On the other hand, if d0 is too large,
the luminance of different images will be discrete. In either case ,viewers would see an unnatural image when they move their head.
Therefore, we should determine the optimal distance.
b0 and h0 are parameters that depend on an application. b0 corresponds to the distance between the user’s eye and the screen on
which the 3D image is projected; h0 is the size of the projectablearea. These parameters should be purpose-designed.
3.3 High Resolution Optical System using Fresnel Lens
In the basic system, there is a trade-off between the resolution andthe size of the projected area. Thus, it is difficult to design an opticalsystem that has a high resolution. To solve this problem, we develop
a novel optical system with higher resolution and a larger projectedarea using a Fresnel concave lens. We call the novel optical system”the improved system.” Figure4 shows the improved system.
A number of images of the content displayed on the LCD aregathered by the projection lenses. The gathered images are projected onto the screen using the Fresnel concave lens. The positionof the focal points is the same as the position of the virtual image
generated by a projection lens. Therefore, the distance between
viewpoints is reduced optically. The following expressions holdtrue for the improved system.
−

1
1
1
+
=
a1 b1
f1

(4)

b1
h1
=
a1
h0

(5)

1
1
1
−
=
a2 b2
f1

(6)

b2
d1
=
a2
d0

(7)

a1 + b2 = b0(8)

a1 is the distance between the position of image of LCD producedby the projection lens and the Fresnel lens. b1 is the distance between the position of image of LCD produced by the Fresnel lensand the Fresnel lens. a2 is the distance between the projection lensand the Fresnel lens. b2 is the distance between the focal points andthe Fresnel lens. f1 is the focal length of the Fresnel lens. h1 is thesize of the actual image of the content on the LCD. d1 is the distance between the viewpoints. When the lenses are arranged suchthat they satisfy the above expressions, the high-resolution opticalsystem can be achieved.
3.4 Projection Methods
As mentioned previously, to superimpose 3D images in real space,
we can choose from two projection methods. Figure5 shows thetwo methods.
In the real object projection method, the retro-reflective screen is
attached to a real object. Then, the image is projected onto the object. The screen shape can be selected depending on the application
requirements. Image correction according to the screen shape is not
required. Therefore, we can design a touch-sensitive soft screen, aRetro-reﬂector

Hand Position
IR Camera
Retro-reflector

LCD

PC

Lens

High Density
Projector Array

Half-mirror
Multi-view Images

12 ~ 20mm

600mm

Half Mirror
Touch Sensation
Haptic Device

Figure 6: System overview.

Figure 7: Experimental condition.

Luminancecomplexly curved screen, or a screen with an automatically moving
surface.
In the floating image method, the retro-reflective screen is arranged as shown in Figure5 and the user cannot see the screen directly. The projected image seems to float in the air when it is seen
through a half mirror. The user can directly touch the 3D image by
wearing a haptic device on his/her hand. This method is suitable forsome applications in which the image is required to move aroundautonomously.

Camera3.5 Input Interface with Haptic Feedback
RePro3D has an intuitive input interface that recognizes the movement of the user’s hand by means of an infrared camera. The camera is located in the projected area, as shown in Figure6, and it captures images of the user’s hand movements. Upon the detection ofcontact between the user’s hand and the projected image from thecaptured image, the system recognizes the degree of contact . According to the recognition result, image position and image posture
are updated.
In addition, the user wears a haptic device on his/her finger.
When the user touches the 3D image, he/she feels a tactile sensation
generated by the haptic device. The mechanism that produces thesensation on a user’s finger is based on a technology called Gravity
Grabber[9]. Gravity Grabber produces fingerpad deformation using a pair of small motors and a belt. To create a ”push” sensation,
the dual motors are driven in opposite directions so that they rollup the belt, thus delivering a vertical stress to the user’s fingerpad.
The belt tension is determined by the degree of contact between thefinger and the 3D image.
4

R ESULTS

4.1 Optimization of the Distance between Viewpoints
As mentioned in Section 3.2, we should determine the optimal distance that results in less crosstalk and smooth motion parallax. We
performed an experiment to determine this distance.
Figure7 shows the experimental setup. We arranged an LCD, ahalf mirror, a retro-reflective screen (Reflite, 8301), and three projection lenses as shown in Figure7. The distance between the viewpoints and the screen was 600 mm. The projection lenses were 12mm in diameter, and their focal length was 48 mm. A white square
was displayed on the LCD, and the size of the square on the retroreflective screen was 100 mm×100 mm. The luminance of the LCD
was 400 cd/m2 .
We placed a digital camera on the straight line on which the focal
points lie. This camera was used to capture the image of the whitesquare projected on the retro-reflective screen. The distance between the projection lenses was varied from 12 mm to 20 mm. We
calculated the average luminance of the pixels in the white square

Ey

eP

ts

oin

os

wp

iti

on

en

ce

tan

e
etw

Vie

b

Dis

Figure 8: Luminance value as a function of spacing between projection lenses.

for each picture. Figure8 shows the relation between the luminancevalue and the spacing between viewpoints.
From this figure, it can be seen that the change in luminance isdirectly proportional to the distance between the viewpoints. Whenthe distance between the viewpoints is in the 12-16 mm range, thechange in luminance is less than 20%.
4.2 Optical Simulation and Implementation
In the improved system, it is difficult to directly calculate the opticalparameters that satisfy the relational expressions of the system. In
addition, we should consider the specifications of available equipment, such as the size of a lens or the resolution of an LCD. Therefore, we wrote an optical simulation program that determines the
optimal parameters (Figure9). We used this simulator to determine
the optical parameters, which are listed in Table1.
On the basis of the simulation results, we built a prototype ofthe system. Figure10 shows the prototype. We used 42 projectionlenses, each of diameter 25 mm and focal length 25 mm. The Fresnel concave lens used in the prototype was 370 mm in diameter andits focal length was -231 mm. We used a high-luminance LCD witha resolution of resolution 1680×1050 pixels and luminance of 1000
cd/m2 . The floating image method was selected as the method of
projection.
The projection lenses were arranged in a 6 (rows) × 7 (columns)
matrix; therefore the total number of viewpoints was 42. The
resolution of the projected image seen from each viewpoint was
175×175 pixels. The device could project up to a distance of 400

Retro-reﬂector
Lens Array
IR Camera

Figure 9: Optical simulation software.

LCD

Table 1: Optical parameters of the prototype device.

f0
a0
b0
d0
f1
a1
b1
a2
b2
d1

125 mm
209 mm
310 mm
30 mm-231 mm
128 mm
284 mm
182 mm
101 mm
16 mm

mm from the user’s viewpoint. The image was projected in a spaceof size 200 × 200 × 300 mm.
Figure11 shows the 3D object that was projected onto the retroreflective screen and can be seen from several viewpoints. The
positional relationship of each displayed objects changes according to the change of viewpoint. This finding shows that our proposed method can produce a stereoscopic image superimposed inreal space with smooth motion parallax.
4.3 Interaction with 3D Image
We placed an infrared camera (Point Gray Research, Firefly MV)
with an IR pass filter and infrared LEDs above the projected area tocapture the user’s hand movements. Then, we implemented the user
input system, which recognizes the degree of contact between theuser’s hand and the displayed image. Using this function, we built
an application that enables the user to touch a character floating inspace. If the user touches the character, the character reacts to theuser’s touch and the user can perceive this reaction by looking at
the changes in the character’s appearance and sound cues. Figure12
shows the application; the character, an animated fairy, floating inreal space, reacts upon being touched by the user’s finger.
Thereafter, we implemented the haptic feedback system. Figure13 shows the result of image processing . The belt tension isdetermined by the degree of contact between the finger and the 3D
image. Figure14 shows the user’s finger with the haptic device onit.
5

D ISCUSSION

RePro3D can generate a 3D image that is superimposed in real
space with smooth motion parallax. However, from a particular

Half Mirror

Fresnel Lens

Figure 10: Prototype of the system.

viewpoint, some level of blur and crosstalk could be seen in the image. At first, we discuss the cause of the blur. When the viewpoint
moves, the image edges appear blurry; however, the blur cannot beseen in the center of the image. One of the reasons to which thisblur can be attributed to is the effect of the angle of incidence oflight rays on the Fresnel concave lens. The angle of incidence increases as we move from the center of the image toward its edges.
In order to decrease such blurring, we should choose a lens that isrobust to changes in the angle of incidence.
Next, we discuss the cause of crosstalk. When a viewpoint
moves back and forth, there are a few points on this path of motion from which several different images can be seen. This problem
is attributed to the retro-reflector performance. The retro-reflector
strongly reflects the light in the incident direction; however, there isa minor diffuse component. Therefore, at a viewpoint that is located
far away from the focal points, several images will appear to bemerged. In Section 4.1, we obtained the optimal distance between
the viewpoints along the vertical and the horizontal directions. In
order to increase the area in which the user can see a more natural 3D image, we should optimize the optical parameters by taking
into consideration the crosstalk that results from the back-and-forthmotion of the viewpoint.
Finally, we discuss the result of the interaction between the 3D
image and the user’s hand. Users can intuitively interact with thevirtual character floating in real space. When the user places his/her
hand into the projected area, the stereoscopic sensation does not disappear. This finding implies that the 3D image is naturally superimposed in real space. However, when the user places his/her handin front of the displayed object, the object was not hidden by theuser’s hand; this is a shortcoming of the system . As future work,
we intend to measure the shape and depth of a user’s hand and display the correct positional relationship between the hand and thedisplayed object .

Figure 11: Projection of 3D object onto the retro-reflective screen as
seen from multiple viewpoints.

Figure 13: Result of image processing for detecting contact between
the 3D image and the user’s hand.

Figure 14: User touching the floating 3D character. Haptic feedback
is provided to the user by the wearable haptic device.

R EFERENCES

Figure 12: Interaction between user’s hand and 3D image.

6

C ONCLUSION

In this paper, we propose a novel full-parallax 3D display system
RePro3D that is suitable for interactive 3D applications with haptic feedback. We designed an optical system consisting of a highdensity projector array; we built the projector array using an LCD
and a number of projection lenses. Then, we used a Fresnel concave lens to improve the resolution of the designed system. Next,
we performed an experiment and an optical simulation to obtain
the optimal parameters of the optical system. On the basis of theresults of the experiment and the simulation, we built a prototypeof the system that could produce 42 parallax images. The prototype
showed that our proposed method can produce stereoscopic images
superimposed in real space with smooth motion parallax. Finally,
we created a user interface that enables users to physically interact
with a virtual character floating in space. As a result, using RePro3D, we created a visual and haptic interface that enables users
to see and touch a virtual 3D object as if the object were real.
ACKNOWLEDGEMENTS
This work was partially supported by CREST of JST (Japan Scienceand Technology Corporation) and JSPS Fellowships.

[1] M. Inami, N. Kawakami, D. Sekiguchi, Y. Yanagida, T. Maeda and
S. Tachi: Visuo-Haptic Display using Head-Mounted Projector, Proceedings of IEEE Virtual Reality 2000, pp.233–240, Mar 2000
[2] G. Lippmann: La photographie integrale, Comptes-Rendus 146, pp.
446.451, 1908
[3] Roberto Lopez-Gulliver, Shunsuke Yoshida, Sumio Yano, Naomi Inoue: gCubik : Real-time Integral Image Rendering for a Cubic 3D
Display, ACM SIGGRAPH 2009 Emerging Technologies, New Orleans, USA, August 3-7 2009[4] Koike, T., Sakai, H., Shibahara, T., Oikawa, M., Yamasaki, M., andUtsugi, K. :Light field copy machine. In ACM SIGGRAPH ASIA
2009 Emerging Technologies: Adaptation (Yokohama, Japan, December 16 - 19, 2009). SIGGRAPH ASIA ’09.
[5] Sakamoto, K.; Kimura, R.; Takaki, M.: Parallax polarizer barrier
stereoscopic 3D display systems, Active Media Technology, 2005.
(AMT 2005). Proceedings of the 2005 International Conference on, pp.469- 474, 19-21 May 2005
[6] Daisuke Kondo, Toshiyuki Goto, Makoto Kouno, Ryugo Kijima andYuzo Takahashi: A Virtual Anatomical Torso for Medical Education
using Free From Image Projection, Proc. of the 10th International
Conference on Virtual Systems and Multimedia, pp678-685, 2004
[7] A. Cassinelli and M. Ishikawa: Khronos projector. In SIGGRAPH
2005, Emerging Technologies, 2005.
[8] Y. Kawaguchi:The Art of Gemotion in Space, Information Visualization, 2006. IV 2006. Tenth International Conference on , vol., no.,pp.658-663, 5-7 July 2006
[9] Kouta Minamizawa, Souichiro Fukamachi, Hiroyuki Kajimoto, Naoki
Kawakami and Susumu Tachi: Gravity Grabber: Wearable Haptic
Display to present Virtual Mass Sensation, 34th Int. Conf. On Computer Graphics and Interactive Techniques (ACM SIGGRAPH 2007),
Emerging Technologies, 2007.