SenseableRays: Opto-Haptic Substitution
for Touch-Enhanced Interactive Spaces

Jun Rekimoto
Interfaculty Initiative inInformation Studies
The University of Tokyo
7-3-1 Hongo, Bunkyo-ku
Tokyo 113-0033 Japanrekimoto@acm.org
Interaction Laboratory
Sony Computer Science
Laboratories, Inc.
3-14-13 Higashigotanda,
Shinagawa-ku
Tokyo 141-0022 Japan

Abstract
This paper proposes a new haptic interaction system
based on optical-haptic substitution. This system
combines time-modulated structured light emitted to
the workspace and a mobile or finger-mounted module
consisting of a photo-detector with a tactile actuator.
Unlike other tactile feedback systems, it does not
require any complicated mechanism for position
sensing and tactile actuation. Instead, it directly
converts time-modulated structured light into haptic
sensations. By sensing this light with a photo detector,
users can feel this time-modulated light as haptic
sensations. The system can easily add haptic feedback
to a wide variety of applications, including surface
computing systems and 3D interactive spaces.

Keywords
Haptic interactions, interactive devices, digital micromirror device, time-modulated structured light

ACM Classification Keywords
H5.2. Information interfaces and presentation (e.g.,
HCI): User Interfaces.
Copyright is held by the author/owner(s).
CHI 2009, April 4 – 9, 2009, Boston, MA, USA

Introduction

ACM 978-1-60558-246-7/09/04.

Haptic sensations play a very important role in physicalinteractions among persons in that the sense of touchenables us to physically communicate with each othersmoothly, rapidly, and accurately. We naturally andextensively use this sense every day during momentsof physical interaction.
Based on this simple fact, several systems have been
proposed for incorporating haptic feedback into humancomputer interactions. While normal graphical userinterfaces (GUIs) mainly rely on visual information,these experimental systems use synthetic haptic
sensations as one facet of human-computerinteractions.
There are two kinds of haptic feedback types. In one
type, real-world physical force is emulated as
accurately as possible. Devices belonging to thiscategory often use mechanical rods [13][15], motorcontrolled strings [8], or other physical mechanisms. In
the other type, sometimes referred to as the “tactile
display” type, physical sensations are created as anadditional feedback method. Examples of devices inthis category include mouse or game-pads withvibrators and touch panels with an actuator that adds atactile "click" feeling to an interaction. Realistic
simulation of the physical force is not always necessary
for devices in this category, but haptic sensations are
effective in telling users when an onscreen button ispressed.
For devices of both types, a user's workspace is greatly
limited because of device architecture. The space for
operating a typical mechanical rod-based forcefeedback system is generally only about 10-30cm2.
Tactile-enhanced touch panels implicitly assume afinger is always in contact with the panel surface. These
devices are therefore not suitable for applications thatfigure 1. The SenseableRays prototype system directly
converts time-modulated light signals emitted from a DMD
projector into haptic sensations, eliminating the need forother complicated position sensors. Haptic feedback can becontrolled at pixel-level resolution.

require larger workspaces, such as manipulating freespace gestures, using sports-related interfaces,computer assisted surgery, and other body-based
activities.
In this paper, we propose a new sensing architecture to
add haptic feedback to interactive space without using
complicated 3D sensors or space-limiting mechanics.
Instead, we combine time-modulated structured light(typically a light emitted from a projector using a digital
micro-mirror device (DMD)), and a piezoelectric
actuator driven by a phototransistor. Figure 1 shows
our prototype system, which we call “SenseableRays”.
The idea of directly converting light signal into tactile
sensations greatly simplifies the system configuration.

89:$!.;<;'(+$=;)4&>=;44&4#$$
24&?,)'&4!

interaction techniques it enables, and future possible
applications.

SenseableRays: Opto-Haptic Substitution@;=,>=&.A+(',.$+;<B'$!1#$'()*+,$2(3,4%$5!
!"#$%&$'()*+,$-,,./()0!

Figure 2 shows the principle of our proposed optohaptic substitution architecture. This sensing/actuation
system consists of a digital micro-mirror device (DMD)
projector that emits time-modulated light rays, and atactile actuator module that can be mounted on a finger,
or can be embedded in a mobile object (Figure 3).

!6#$'()*+,$2(3,4%$7!

figure 2. The opto-haptic substitution principle. The
combination of digital micro-mirror device (DMD)
projector that emits time-modulated spatial light and afinger mounted photo-transistor and actuator createsspatial tactile feedback without using complicated 3D

The light wavelength used by a projector can be either
visible or invisible (i.e., infrared). Our current prototype
uses visible light emitted from a commercially available
DMD projector without any modification. The projector
uses a micro-mirror array to time-modulate the light ofeach pixel, and in combination with a circulating redgreen-blue filter, it creates arbitrary color andbrightness for each pixel. As a result, each color has its
own (unique) time-modulated pattern. By sensing thispattern with a photo sensor, it becomes possible tovibration pattern on a tactile module.

position sensors or mechanical force-feedback
actuators.

It eliminates the need for complicated and slow position
sensing. Since DMD projector light color and intensity
can be controlled pixel-by-pixel, light emitting timepatterns for each pixel can also be controlled, because
a DMD projector creates arbitrary colored pixels bychanging mirror on-off timing at each pixel. This means
haptic feedback can be created and controlled at the
projected pixel resolution. We call this combination“opto-haptic substitution” because it translates timemodulated light signals into haptic sensations. In thispaper, we present our prototype system, theFigure 4 shows two variations for connecting timemodulated light with tactile stimulation. In one (typeA), light is regarded as a coded signal and a tactile
module decodes its code and triggers a corresponding
vibration pattern. In the other, and simpler, variation(type-B), time-modulated light itself is used as avibration pattern. The latter approach would make atactile module significantly simpler. It would merely
require a phototransistor and an amplifier to drive atactile actuator.
A tactile-feedback module can be attached to various
kinds of input devices. If it is combined with a physicalicon (or “phyicon” [11]), it supports tactile-enhanced
interaction. We are interested in attaching it to a finger
to directly stimulate the finger. We attach a tactilefeedback unit to the fingernail so that the latter does
not hinder normal finger interactions (Figure 3 left).
The number of tactile modules:
Unlike most other previous tactile feedback systems,
there is no technical limitation to increasing the numberof tactile units. Each unit receives casted light cast at
each position, and may have its own tactile vibration
based on the received light signals. This feature makes
it possible to create applications using multiple input
devices, multiple fingers, and multiple users. In theexample shown in Figure 7, the user has two units
attached to his/her hand, one to the index finger andthe other to the thumb. This enables the user to sense
the size of virtual objects.

48,&9:!

48,&9;!

%012!'301/"%!

%012!'301/"%!

)&#5)&!
6676776!

4"#$%&',"-&./'#5)&*55+!

!"#$%&'(&&)*"#+',"-&./!

!"#$%&'(&&)*"#+',"-&./!

figure 4. Two ways of translating light into tactile
sensations. Type-A uses emitted light as an index tovibration pattern dictionary. Type-B simply activates anactuator by means of amplified received light signal.

Comparison with position sensing methods:
Although it is also possible to combine a 3D position
sensor and a tactile unit, such as is common in some
virtual reality systems; we believe the opto-haptic
approach has several advantages over this approach.

!"#$#%$&'()*)$#&!
'-!.*/0&!

!"#$%&'(!

)*+!

!"#$#%$&'()*)$#&!

345#.6789:1!

,*$$"-#.)#/#01!

2'%(*!

'+$,'$#&!
'+$,'$#&!
figure 3. Modules that receive and translate light signal
into tactile feedback. They can be embedded in various
input devices or it can also be mounted on a finger.

figure 5. Light signals sensed by a tactile module.

-+'./($0'1*2"$3/($0*3403$+*5'34.6*'""+$'),!
78*9$3/($0*:403/0;!

"$3/($0!

:414)#*-')(14*9'<4+0!

-+/;;4+*=)#>'#$+!

-')(14*?44.5')@!

2BC%7CD3*.41'E6!

!"#$%&'"()*'""+$'),*!
A/;,#*:403/0;!

-+/;;4+*=)#>'#$+!

-')(14*?44.5')@!

figure 6. Comparison between opto-haptic approach andthe other approach using traditional 3D position sensing.

One is its simplicity. Currently available 3D sensors,
such as those based on magneto-electric or ultrasonicsensing, require complicated position estimation
method. In addition, using this approach makes theresulting tactile unit large and heavy and thus it cannot
be comfortably mounted on a fingertip. In contrast, theopto-haptic approach does not explicitly sense positions
and only requires a photo-detector for signal detection.
Another advantage is response time. To create a goodtactile feedback system, the delay between motion andfeedback must be as short as possible. Currently
existing 3D sensors such as magnetic-electric sensors
have a 10-30 ms delay, which degrades the quality oftactile interaction. However, the opt-haptic approach,especially the Type-B approach, is quite simple andmerely connects light modulation to vibration (Figure 6).
There is virtually no sensing delay or communication
delay between sensing and stimulation (which is lessthan few microseconds). As a result, the delay time
becomes virtually negligible.

figure 7. Feeling the size of the projected object.

The third advantage is the ease of creating applications.
Application can define arbitrary “tactile” pattern bysimply painting the corresponding region with a specificcolor or visual patterns.

Basic Interactions
Using this architecture, we expect the following three
haptic-enhanced interactions to become possible.
Defining target position: If a tactile area surrounds atarget object, such as an icon, users can easily know
when their finger is within proximity of the target.
Using invisible rays also makes it possible to inform theuser of a particular zone of a position in free space.
Defining object boundaries: Object shapes can also
create tactile boundaries tat users can feel thisboundary by both visual and tactile feedback.

Defining tactile texture: Projecting areas by different
light-pattern causes a different time-modulated pattern
for each area. This differences can be felt though tactile
sensation.
These interaction techniques are not limited to
onscreen (projected images) interactions. It is easy to
add tactile above tactile interaction to real world
objects, by just projecting time-modulated light tothem.

The Prototype System
Figure 8 shows an implementation of our current
finger-mounted tactile module. This is based on the“Type-B” approach described in the previous section(i.e., an actuator is directly driven by received lightsignals). The finger-mounted sensing circuit isextremely simple, consisting merely of aphototransistor and an operational amplifier
to drive a tactile actuator.

micro-mirror device (DMD) to control projection light. A
finger-mounted photo-sensor detects time-modulated
light. As shown in the graphs in Figure 5, light-patternchanged according to the corresponding light's colorand luminance. The frequency of time-modulation isabout 125Hz. This frequency is also adequate forproviding tactile stimulation. As a whole, the device
enables resulting tactile feedback to be very clearly
sensed by a finger.
We are also interested in introducing DMD-Kit [10], a
programmable digital micro-mirror device system, into
our system. While off-the-shelf projectors are sufficientfor providing simple tactile feedback, it is not easy tocreate various tactile patterns with them, because
time-modulation patterns are predefined. For example,
corresponding light patterns of red and blue pixels

We choose the TouchEngine [16] as ourtactile actuator. It uses a multi-layered
piezoelectric ceramic to create physicalmotion. Unlike other actuators, such as those
based on small motors or magnet coils, thedelay time required to initiate physical motion
is very short (i.e., less than 2.5ms, which isthe delay due to the piezoelectric actuator,and not due to the light sensing). It is also
thin (500 micrometer) so that the entire
sensor-actuator module can be attached to
the fingernail.
We use the off-the-shelf BenQ MP721 PC
projector as a light source. It uses a digital

figure 8. The SenseableRays system in use.

provide almost identical haptic feeling, because only
the phase of the signals are different. Using a
programmable DMD system instead would make itpossible to create various types of tactile symbols thatare more clearly distinguishable.
Figure 8 also shows various experimental tactile
patterns we have used experimentally the
SenseableRays system. Since these patterns aretranslated into time-modulated light emission at eachpixel, no additional programming is required to various
types of tactile feedback. We also tried using normal
computer screen images showing GUI windows andother WIMP objects. In so doing we found that these
GUI objects also create tactile feedback to the fingers.
This suggests that our system has the potential to add
tactile feedback to existing GUI applications without theneed for any serious modifications. This feature shouldbe very useful for assisting people with visualdisabilities.

Potential Applications
Using this system’s architecture will enable us to createa various kinds of interactive systems. Below are someexamples of the types of interactions that we believe
the architecture will make technically feasible anduseful.
Tactile feedback for free space interaction:
Recently, multi-touch or gesture-based interactions are
becoming increasingly common. Systems supportingthese interactions are able to detect finger positions bymeans of various sensing technologies, but do notprovide tactile feedback. If the fingers are not in
contact with any surface, such as in the interactions

figure 9. SenseableRays is used for Surgery
supportdepicted in the movie “Minority Report”, tactile
feedback is difficult to provide. Our opto-haptic system
can provide a solution to this problem.
Sports Training:
Most if not all, sports require precise body control orequipment control on the part of those who participate
in them. For example, in sports such as tennis or golf
where the use of specific instruments is required,precise control in moving these instruments is often the
key to training in these sports. In 3D space, it is no
insignificant matter to know precisely how these
instruments are being moved. Our opto-haptic system
can help to achieve this. That is, if training space has
been illuminated by time-modulated structured light,
and sports instruments have an opto-haptic
substitution module, training participants can easily
recognize whether the instrument is being correctly
moved by feeling its tactile feedback.

Medical Support:
Surgery is another domain where precise haptic
sensations can provide important support. For example,
as illustrated in Figure 9, an opto-haptic unit can be
attached to the surgeon's knife. This enables thesurgeon to feel the “cut-path” as a haptic sensation bymeans of a light line projected onto the patient’s body.
This cut-path information can also be applied to othermedical sensing technologies, such as ultrasonicsensing or magnetic resonance imaging (MRI).
Navigation Support:
Opto-haptic substitution can also be used as anavigation method for navigating one’s way throughlarge buildings or shopping malls. If each corridor orspot in such an environment is illuminated by a uniquetime-modulated light, a small handheld device canreceive and decode such light signals and translatethem into tactile feedback. Such devices do not need tohave a display and thus it can be much smaller thanexisting mobile devices or cellular phones.

Extending to 3D Haptic Space
Our current prototype uses a single projector as a lightsource. Thus a defined tactile feedback area is limited
to 2D. We consider this 2D tactile area to be useful formany potential applications such as multiple-finger
gesture systems or surgery support systems. However,for applications that require true 3D tactile feedback
area, we are also planning to integrate two or more
light sources. By combining two projectors are
combined as shown in Figure 10, we can make an optohaptic vibration unit that triggers tactile feedback only
when a cross section of lights from both projectors is
obtained. In this case, we use the Type-A variation
shown in Figure 4 and the projectors are synchronized

figure 10. Defying 3D tactile spot by combining
two light sources

so that their light emissions will not overlap. Then aphoto-sensing part determines which projector is thelight source by decoding the received light. The device
will activate tactile feedback only when it receives lightfrom both projectors. For example, when the firstprojector emits a line pattern and the second projector
emits a circle pattern, the resulting cross-section
pattern becomes a circle in the air. In this case a finger
unit will activate tactile feedback only when the finger
comes into contact with this circle.

Related Work
Vision Substation Systems:
The idea of using tactile sensations to feel visual
images has a long history, especially in substituting
these sensations for vision for people with visualdisabilities. The device known as Optacon, developed in
the early 1960s, was probably the earliest example ofthis kind. This device has a 6 x 24 matrix of mechanicalpins vibrating at about 230 Hz. It directly maps thebrininess input from a camera to corresponding pins.
Another well-known example is Collins's Tactile Vision
Substitution System (TVSS) [4], which was developedin 1970. The TVSS uses a 20 x 20 matrix of actuators
in contact with a 10-inch square of skin. Since then,many similar systems have been developed. Recent
examples include Kajimoto et al's ``Forehead electrotactile display'' [17]. It is a vision substitution systemthat is mounted on a user’s forehead.
SmartFinger [3] is a fingernail-mounted tactile device
that stimulates ones finger with vibration. As in oursystem, vibration is controlled by a photo-detector thatis also mounted on a fingernail. This photo-detector
senses physical surface texture, or in other applications,the density of ink printed on a paper. FingerSight [7] isa device that combines a finger-mounted camera and asmall cell-phone vibrator. Toshio Iwai's interactive
artwork “SOUND-LENS” is a hand held device thatconverts light into sound. These systems inspired us tomake a nail-mounted haptic device with timemodulated structured light space. While all of the above
systems merely use natural images as input, oursystem also uses time-modulated structured light thatmakes tactile feedback more controllable and suitablefor various applications.
Tactile Actuators:
Many researches and manufactures have developed
touch-panels with haptic feedback [6][16]. As an
example of a related product, the input device company
ALPS has developed “Force Reactor”, a vibrator using
an electromagnet, which is widely used to add motion
feedback to cellular phones [2]. These products, which
help users to feel displayed objects both by visual andtactile feedback, require a user to actually touch thesurface and relying on vibration of the panel itself. Thisconfiguration is suitable for small touch panels, such as
mobile-phone displays. On the other hand, applications
that require a larger workspace are not covered bythese touch-panels. Yao et al. designed an enhanced
surgery probe that enhances the feeling of the probe byamplifying added motion to it. Although its primary
purpose is very different from ours, it shows how
structured sensing light can further enhance surgery
probes, as described in the previous section.
Another very different method of creating haptic
sensations in free space has recently been developed[12]. This method is an extension of “spotlight audio”,
a technique for localizing or directionalizing audio
signals by using an array of transducers. With thismethod, lowering the signal frequency enables people
to feel haptic sensations instead of sound. Its main
advantage is that no finger-mounted unit is required.
Although this method is very interesting and has
potential, the parameters that have currently beenachieved for it (e.g., accuracy, sensing delay,transducer-hand distance) are still not on a sufficiently
high level to create realistic applications.
Using DMD for Interaction:
Recently, several researches are interested in using
DMD as a method for structured light source andsensing. For example, Lee et al.’s system [14] uses
DMD for position sensing, and Cotting et al.’s system[5] uses it for detecting 3D shapes of screens. However,none of the systems developed so far use it for
generating tactile sensations. In this sense, our system
is very different from these previous systems because it
enables positional tactile feedback in the air to beprovided by using structured light.

Conclusions
In this paper, we proposed a system we call“SenseableRays” that enables generation of tactileenhanced interactive space without requiring
cumbersome mechanical rods or other tethered
solutions. Our approach combines time-modulated
structured light with a simple photo-sensing actuator,and makes it quite easy to define tactile areas within
any area of interactive space. We also suggested
various potential applications for it, including sports
training and surgery support. Our current plans are toextend this system to incorporate 3D interactions into it.

References[1] Artificial muscle home.
http://www.artificialmuscle.net.
[2] ALPS. Alps Electric Press Release Force Reactor
http://www.alps.com/e/news release/2005/0608
01.html, 2008.
[3] Hideyuki Ando, Takeshi Miki, Masahiko Inami, andTaro Maeda. Nail-mounted tactile display. SIGGRAPH
2002 conference abstracts and applications, (2002) 7878.
[4] C.C. Collins. Tactile television - mechanical andelectrical image projection. IEEE Trans. Man-Machine
Systems, (1): (1970) 65-71.
[5] D. Cotting, M. Naef, M. Gross, and H. Fuchs,
Embedding Imperceptible Patterns into Projected
Images for Simultaneous Acquision and Display, Third
IEEE and ACM Symposium on Mixed and Augmented
Reality (ISMAR) (2004) 100-109.
[6] Masaaki Fukumoto and Toshiaki Sugimura.
ActiveClick: Tactile feedback for touch panels. In CHI
2001 summary, (2001) 121-122.
[7] John Galeotti, Samantha Horvath, Roberta Klatzky,
Brock Nichol, Mel Siegel, and George Stetten.
Fingersight: fingertip control and haptic sensing of thevisual environment. SIGGRAPH 2008 new tech demos,(2008) 1-1.
[8] Yukihiro Hirata and Makoto Sato. 3-dimensional
interface device for virtual work space. In IEEE/RSJ
International Conference on Intelligent Robots andSystems, (1992) 889-896.
[9]

IMMERSION. Cybertouch.

[10] Texas Instruments. DLP Discovery
http://focus.ti.com/dlpdmd/docs.
[11] Hiroshi Ishii and Brygg Ullmer. Tangible bits:
towards seamless interfaces between people, bits andatoms. Proceedings of CHI97, (1997) 234-241.
[12] T. Iwamoto, M. Tatezono, and H. Shinoda, NonContact Method for Producing Tactile Sensation Using
Airborne Ultrasound, Proc. EuroHaptics 2008, Springer
LNCS vol.5024 (2008) 504-513.
[13] Hiroo Iwata. Artificial reality with force-feedback:development of desktop virtual space with compact
master manipulator. SIGGRAPH Comput. Graph.,24(4): (1990) 165-170.
[14] J.Lee, S.Hudson, J.Summet, and P.Dietz, Moveable
Interactive Projected Displays Using Projector Based
Tracking, ACM UIST, (2005) 63-72.
[15] Margaret Minsky, Ouh young Ming, Oliver Steele, Jr.
Frederick P. Brooks, and Max Behensky. Feeling andseeing: issues in force display. SIGGRAPH Comput.
Graph., 24(2): (1990) 235-241.
[16] Ivan Poupyrev, Shigeaki Maruyama, and Jun
Rekimoto. Ambient touch: designing tactile interfaces
for handheld devices. ACM UIST 2002, (2002) 51-60.
[17] H.Kajimoto, Y. Kanno, S. Tachi. Forehead electrotactile display for vision substitution. EuroHaptics, 2006.
H.Yao,V.Hayward,and R.Ellis, A Tactile Enhancement
Instrument for Minimally Invasive Surgery, Computer
Aided Surgery, 10(4), (2005) 223-239.