ARTICLE IN PRESSInteracting with Computers xxx (2008) xxx–xxx

Contents lists available at ScienceDirect

Interacting with Computers
journal homepage: www.elsevier.com/locate/intcom

Tactile sensory substitution: Models for enaction in HCIYon Visell *McGill University, Deparment of Electrical and Computer Engineering, Centre for Intelligent Machines and CIRMMT, 3480 University Street, Room 424,
Montreal, Que., Canada H3A 2A7

a r t i c l e

i n f oArticle history:
Received 1 April 2008
Received in revised form 2 July 2008
Accepted 8 August 2008
Available online xxxx

a b s t r a c t

Keywords:
Enaction
Tactile display
Sensory substitution
Enactive interfaces

To apply enactive principles within human–computer interaction poses interesting challenges to the waythat we design and evaluate interfaces, particularly those that possess a strong sensorimotor character.
This article surveys the ﬁeld of tactile sensory substitution, an area of science and engineering that liesat the intersection of such research domains as neuroscience, haptics, and sensory prosthetics. It isargued that this area of research is of high relevance to the design and understanding of enactive interfaces that make use of touch, and is also a fertile arena for revealing fundamental issues at stake in thedesign and implementation of enactive interfaces, ranging from engineering, to human sensory physiology, and the function and plasticity of perception. A survey of these questions is provided, alongside a
range of current and historical examples.
Ó 2008 Elsevier B.V. All rights reserved.

1. Introduction

1.1. Tactile feedback in enactive interfaces

In tandem with contemporary advances in information technologies, there has been considerable interest in enriching interactive
computing systems so as to make more effective use of theperceptual and motor capabilities of their users. Applications ofcurrent interest include robotic microsurgery devices, immersive
virtual environment (VE) simulations, tangible information systems and gesturally controlled video games. The many interfaces
and interactions exempliﬁed within the state-of-the-art depend
in turn upon a wide range of continuous input technologies. The
scope of this design space serves to highlight the broad importanceof enactive considerations for HCI design today. As other contributors to this issue have noted, the enactive viewpoint emphasizes
the strong supportive roles that can be played by the coordinatedaffordance of action and perception in interactive systems, andby the good use of existing sensorimotor abilities on the part ofusers. Such considerations may be crucial for ensuring that continuous, movement-based control affordances attain the degree ofusability and intuitiveness that they are capable of. Sensory substitution devices (introduced below) provide a minimal yet highly
illustrative class of examples for the diverse technological and human factors considerations that must be accounted for when
designing for such interactions. The purpose of this article is to review these considerations in the context of tactile sensory
substitution.

An emphasis on tactile feedback in interaction lends context tothe present review, and exempliﬁes a compelling parallel thread ofdevelopments in HCI, centered on touch-based interfaces. Tactile
information channels play a signiﬁcant role in diverse everyday
interactions. Since most human–computer interaction is mediatedthrough touch, the tactile sense should be positioned to play aricher role in such interactions than it does at present. However,guidelines for the design of non-visual displays remain signiﬁcantly less developed than in the visual case, and case studies arefewer. Consequently, added attention to this area is warranted.
Because of the knowledge gap that exists relative to visual (andto some extent auditory) display design, tactile display is arguably
under used. Moreover, tactile display possesses distinct advantages
relative to other modalities, while also posing unique challenges tothe interface engineer and designer. These issues are reviewed
beginning in Section 2. As noted above, many of them are ofbroader signiﬁcance to enactive interaction design.

* Tel.: +1 514 967 1648; fax: +1 514 398 7348.
E-mail address: yon@cim.mcgill.ca

1.2. Sensory substitution
Sensory substitution refers to the translation of sensory
information that is normally available via one sense to another.
Occasionally, the term is also used to refer to plasticity in the
intrinsic senses, i.e. to cases in which an intrinsic sensory capacity
is used for a purpose normally tied to another such capacity. The
best studied example is the phenomenon of ‘‘blindsight” by visually impaired people, in which heightened use of auditory channels
enables individuals using it to discern objects, obstacles or0953-5438/$ - see front matter Ó 2008 Elsevier B.V. All rights reserved.
doi:10.1016/j.intcom.2008.08.004

Please cite this article in press as: Visell, Y., Tactile sensory substitution: Models for enaction in HCI, Interact. Comput. (2008),
doi:10.1016/j.intcom.2008.08.004

ARTICLE IN PRESS
2

Y. Visell / Interacting with Computers xxx (2008) xxx–xxx

structures in front of them (Weiskrantz, 1998). However, morecommonly, and in the sense intended in the current article, theterm is invoked to describe a technological intervention via a system capable of performing such a translation for its user. In suchcases, the electronic sensing capabilities of the device are used toperform the requisite capture and translation of signals. As earlyas 1897, Noiszewski created a device called the Elektroftalm, consisting of an ambient light sensitive selenium cell connected to asound-producing circuit, designed to aid visually impaired people
in distinguishing light and dark spaces (Spirkovska, 2005; Capp
and Picton, 2000). Later versions of the device were designed toutilize 80-sensor arrays and tactile, rather than auditory, feedback(Starkiewicz and Kuliszewski, 1963). Sensory substitution systems
were subsequently proposed for teleoperation settings, in order toassist humans in the remote operation of a robotic manipulator,
where the operator was thought likely to beneﬁt from additional
sensory information. Such feedback has been proposed in the formof tactile or auditory display of contact information to supplementa video display. As discussed below, sensory substitution devices
have also been developed to test fundamental scientiﬁc ideas
about perceptual plasticity, or sensorimotor theories of perception.
1.2.1. Relevance to human computer interaction
Human–computer interaction can be regarded as the design ofsystems that enable their users to interact with digital information,accessed via computers, that would not otherwise be accessible.
That information might be organized by a desktop metaphor,virtual reality simulation, or the abstract space of a dataset to be
explored. The interface operates by extending the user’s perceptual-motor capabilities ‘‘into” these virtual environments, in somesense. Without this sensory feedback, users are impaired in acting
within the virtual environment. From such a standpoint, sensory
substitution can be seen as an organizing perspective for human–
computer interaction, most relevant to continuous (as opposed tosymbolic) methods of interaction. This viewpoint has practicalvalue to the extent that research on sensory substitution is able
to inform the design of new interfaces.
1.2.2. Systems for sensory substitution
A sensory substitution system can be thought of as composed ofa number of components of the kind illustrated in Fig. 1. Information about the environment is typically acquired from sensors
corresponding to modality A, and the information is transduced
into a set of signals x(t) that are subsequently digitized. The
sensors can be physical devices or they may correspond tomeasurements in a virtual environment. A coupling device maps
the sensed data x onto a set of signals y(t) for driving the actuatorsof the display. The actuated display presents the information to ahuman sensory modality B, which is eventually transduced andprocessed by the intrinsic sensory system of the body. In the

Movementof deviceBody

systems of most interest to this article, B will represent a tactile
sensory channel or channels.
One feature (discussed in Section 5.6) that many have argued is
crucial to the effectiveness of such systems is that the interaction
loop is closed, through the affordance of user control over the position and orientation of the sensors, represented by the dashed lines
in the ﬁgure.
1.2.3. Deﬁnitions
As suggested in the loose typology alluded to above, authors
writing about the subject have differed in the level of signiﬁcancethey have attached to both the term itself and to the phenomenology that can be ascribed to the interventions it labels.
1.2.3.1. Operational deﬁnitions. In the narrowest sense, sensory substitution has been taken to refer to the act of translating signalsthat are normally associated with sensory modality A to signalsthat can be detected via modality B. For example, ambient lightintensity may be translated to the vibration of an object attachedto the ﬁngertip, without any pretense of the result constituting avisual or informational aid. Such an operational description avoids
any claim as to the utility of the translated information for a particular task, function, or capacity.
1.2.3.2. Functional characterizations. Various authors have applied
the term to imply the provision of additional sensory information
through a device in such a way as to aid users in performing sometask. More speciﬁcally, some have identiﬁed sensory substitution
as the provision of information to assist people with sensory impairments. For example, several systems have been designed to enable
their users to read printed text, other devices have been created as
navigational aids for visually impaired people, and a number ofsystems have been developed to improve teleoperated control overa remote robot. Examples from each of these categories arereviewed in the sections below.
1.2.3.3. Perceptual claims. Several authors have described sensory
substitution systems as enabling a form of artiﬁcial perception
via the newly provided sensory information (after an initial periodof adaptation). In such cases the intervention may be motivated asa sensory replacement for individuals with visual or auditory
impairments. When such interventions are claimed to be successful, perception is sometimes hypothesized to occur in a cross-modal sense, in which intrinsic perceptual faculties are recruited toutilize the substituted information. As discussed below, support
for these ideas has come from behavioral evidence gathered fromexperiments with users of the systems, from neurological data
about activity in the brain, and from qualitative feedback provided
by users. Ultimately, the available evidence may support diverging
models of what it could mean to acquire artiﬁcial perception
through a sensory substitution device. The scope of the discussion
regarding cross-modal perception somewhat exceeds the domainof the present article, but some of the most signiﬁcant issues are
discussed in Section 5.3 below.
1.3. Tactile sensory substitution

Sensors(Modality A)x(t)Coupling
Devicey(t)Actuators(Modality B)

At an operational level, tactile sensory substitution can be described as the display of phenomena typically associated with
one human sense by means of the tactile sense.1 It poses engineering problems associated with electronic sensing, the coupling ofsensing to tactile display methods, to the design of the displays

Environment

Fig. 1. Structure of a sensory substitution system.

1
The author explicitly allows for the possibility of tactile-to-tactile sensory
substitution, examples of which will be discussed later in this article.

Please cite this article in press as: Visell, Y., Tactile sensory substitution: Models for enaction in HCI, Interact. Comput. (2008),
doi:10.1016/j.intcom.2008.08.004

ARTICLE IN PRESS
Y. Visell / Interacting with Computers xxx (2008) xxx–xxx

themselves, and the presentation of information so as to best takeadvantage of the intrinsic human tactile information processing
pathways. It is specially linked to fundamental questions concerning
the plasticity of human perception and to the ways in which perception is intertwined with physical embodiment, the structure of theenvironment and the means of action. Consideration of these issues
is indispensable to a discussion of how tactile sensory substitution(TSS) devices may be successfully realized. Many applications ofsuch devices have been proposed, ranging from sensory replacementin the case of speciﬁc sensory deﬁcits (such as blindness), to therehabilitation of motor function, and the display of task salient channels of ecological information during human interaction with virtualenvironments. These topics are reviewed in the sections that follow,
with reference to the relevant literature.
1.3.1. Tactile vision sensory substitution
An early example of tactile sensory substitution is provided by
the Tactile Vision Sensory Substitution (TVSS) system, created by
Bach-y-Rita and his collaborators. This research is overviewed in a
recent article (Bach-y-Rita, 2004). The name TVSS refers to a family of devices developed beginning in the early 1960s for the purpose of lending sight to visually impaired people (Bach-y-Rita,
2003). Photographs of two of the systems are shown in Fig. 2.
The systems captured video of their user’s surroundings via acamera, and supply a tactile display of a representation of thevideo information. Several tactile displays were investigated,consisting of two-dimensional arrays of vibrotactile or
low-frequency displacement actuators applied to the skin of theback, forehead, thigh, abdomen, or ﬁnger (Bach-y-Rita, 2004).
The size of the array has varied from 64 to about 1000 elements.
The TVSS has been the subject of an unusual number of studies,over several decades, and has previously been marketed commercially under the VideoTact name, beginning in the 1990s. TVSS
researchers have reported that users of these systems are able
to learn to discriminate simple tactile stimuli, such as orientedlines. Granted further experience with the device, and, in mostcases, the ability to move the camera, users are said to begin torecognize shapes of simple, familiar objects via the tactile display.
After a period of about 15 h, Bach-y-Rita and his colleagues statethat users’ awareness of the proximal tactile sensation fades, andthat users begin to ‘‘perceive” stable three-dimensional objects in3

space (Bach-y-Rita, 2004). This progression is said to occur without conscious processing.
The authors report that users with sensory impairments havelearned to perform tasks including batting a ball, industrial assembly line work, and navigation on foot (Bach-y-Rita, 2004). The
authors emphasize the insensitivity of their results to sensor andactuator location (discussed below). They have broadly attributed
their results with the TVSS to the plasticity of the human perceptual capacity in adapting to the information supplied by the substitution device, and have emphasized the relation to prior results onneural plasticity (some of which are reviewed in Section 5.3,
below). It is suggested that the cross-modal processing capabilitiesof the brain are critical in enabling such substitutions, particularly
in visually impaired people.
2. Tactile display design
The tactile information display is a key component of a tactile
sensory substitution (TSS) device. The research and developmentof such displays is a large subject in its own right, ranging in focus
from the engineering of actuators to the study of tactile psychophysics. In this section, key issues in tactile display are surveyedfrom the viewpoint of sensory substitution.
2.1. Advantages of tactile feedback
Interest in tactile displays has undoubtedly been ampliﬁed in recent years for the same reasons that interest in electronic display
modalities for all of the senses has grown. Although the most common methods for tactile display have been in use for many years,
technological advances in manufacturing, microprocessor and sensor technology during the last decades have made the active components of certain of these systems smaller and less costly. New tactile
displays can be made smaller and higher resolution than has beenpossible in the past, and such devices are now easier to engineer.
This, in turn, has opened a larger range of potential applications
for enrichment by a wider array of technologies that can be applied
to them. A number of recent reviews exist to guide the application
developer in the use of tactile display technologies in human computer interaction (Jones and Sarter, 2008; Wall and Brewster,
2006; Benali-Khoudja et al., 2004; Chouvardas et al., 2005; Hafez,2007).

Fig. 2. Left: This version of the Tactile Vision Sensory Substitution system of Bach-y-Rita and his colleagues (image reproduced from (Bach-y-Rita et al., 1969) consisted of a 20
by 20 element vibrotactile pin array that was applied to the back, by means of the dental chair shown, and was used for displaying imagery from the video camera. Right: A
later version of the TVSS utilized a pin array worn on the abdomen and a video camera worn on the head.

Please cite this article in press as: Visell, Y., Tactile sensory substitution: Models for enaction in HCI, Interact. Comput. (2008),
doi:10.1016/j.intcom.2008.08.004

ARTICLE IN PRESS
4

Y. Visell / Interacting with Computers xxx (2008) xxx–xxx

Tactile feedback possesses qualities that make it more suitable
under many conditions than other display methods. The actuators
involved are cheap and are available in many different forms,
including solenoids, eccentric inertial motors, voice coils, piezoelectric transducers, pneumatic components, and others; see Section 2.3 below. They can be made small enough to match thereceptive ﬁeld sizes of the mechanoreceptors in the most sensitive
parts of the body, and powerful enough to saturate the activationlevels of the same receptors. In this respect, they are well suitedeven to applications involving inexpensive everyday or portabledevices, where they are already seeing widespread use in mobile
phones and video games.
Tactile display requires little power, and is readily made portable. By contrast, traditional force-based haptic displays, which provide tactile information as well as kinesthetic information
associated with the change in forces over distances of more thana few millimeters, require much more space and power to operate.
Furthermore, tactile displays do not possess the instabilities thatforce-based haptic interfaces must be carefully designed to avoid(Adams and Hannaford, 1999). Thus, tactile displays are easierand more cost-effective to produce than haptic displays. Thesebeneﬁts of tactile display have led several research teams to integrate separate tactile display channels with traditional force feedback haptic interfaces (examples Kontarinis and Howe, 1995;
Massimino, 1991; Massimino, 1995 are reviewed in the sections
that follow). Such integrated devices may prove relevant to situations, such as robotic surgery, in which haptic feedback may not
be an option (Kitagawa et al., 2005; Akinbiyi et al., 2006). Many
sources of environmental information are multisensory in nature,
and where physical contact is involved, tactile sensation can play
an important role, as in the case of the perception of surface texture in real or virtual environments.
Tactile display is well suited to presenting ﬁne textural
information to the skin, such as is encountered during manual
exploration of a virtual textile or other surface texture. Touch alone
is enough to allow us to discriminate between surface textures thatdiffer only microscopically. In the experiments of LaMotte andWhitehouse, for example, the detection thresholds of the ﬁngerpad
of macaque monkeys to microdots just a few microns high on anotherwise smooth surface were charted (LaMotte and Whitehouse,1986).
Tactile feedback plays a special role in skilled manipulation
tasks, which normally require touch. Touch is most acute at theinterface between the body and the artifacts that are involved insuch tasks, and it conveys signiﬁcant information about contacts
between the artifacts concerned. This information is complementary to that which is obtained about kinematic constraints via kinesthetic channels (forces, extensions and joint angles).
Equally signiﬁcantly, interest in informational display devices
has grown due to the proliferation of digital information, whether
pertaining to virtual environments or to the state of the real world.
As noted in the introduction, when interacting with virtual envi-

ronments, all users are confronted with a sensory deﬁcit that must
be overcome through the design of sensory displays.
From a cognitive standpoint, touch is often thought of as providing a ground truth to experience – which is why we haveexpressions like ‘‘hands-on demonstration” and ‘‘hands-on activity”. Hearing is, by comparison, notoriously susceptible to confusion (for example, spatial direction and identity can be confused
depending on the stimulus properties and direction). Similarly,
we become aware of the possibility of visual illusions from a young
age. While comparable tactile illusions exist, we rely on touch toreassure ourselves of the reality and validity (or otherwise) of what
we are experiencing.
Applications of human computer interaction that are well
suited for tactile sensory display are diverse. Moreover, tactile
feedback may have a special role to play in motor learning. For
example, light (non-force supplying) touch has been shown tofacilitate postural correction in dynamic balance (Jeka and Lackner,
1994), providing an additional feedback channel to assist thevestibular system. In addition, several systems have been developed to aid motor learning of dynamic gestural tasks involving
control of the body, through the use of vibrotactile feedback (awearable sleeve developed by Lieberman (Lieberman and Breazeal,2007) is a recent example).
2.2. Human tactile sensation
Human skin facilitates sensation through touch via the mechanical properties of its tissues, which are responsible for conveying
tactile stimuli to the dermal and subdermal receptor sites at which
they are transduced into neural signals, and through the tactile
receptors that perform the transduction. Human skin can be categorized as hairy (skin on most parts of the body), glabrous (thenon-hairy skin on the front of the hands and bottom of the feet),or mucous (the moist skin of the mouth and other bodily openings). The skin contains a range of receptors for pain (nociceptors),
temperature (thermoreceptors), chemical stimuli (chemoreceptors), limb joint and muscular states (proprioceptors), and six kindsof tactile force-, or mechanoreceptors (Kandel et al., 2000). Details
concerning the nature, distribution, and properties of such receptors are still open issues. Properties of ﬁve types of mechanoreceptors present in the cutaneous and subcutaneous layers of the skin,and associated connective tissue, are summarized in Table 1.
Detailed descriptions of the anatomy and physiology of receptor
populations in the hands are available, and summarized instandard references on neuroscience (Kandel et al., 2000). The
literature on properties of tactile receptor populations in skin onother parts of the body is less expansive.
In addition, many relevant psychophysical properties are
known. These include the relevant ranges of frequency sensitivity,
receptive ﬁeld sizes, receptor densities, and sensory correlates, allquantities that are highly salient to the low-level design of tactile
displays. Spatial tactile resolution is among the most extensively
studied features. The most frequently cited measure is the two-

Table 1
An understanding of skin tactile mechanoreceptor properties is a necessary prerequisite for the design of effective tactile displays (table adapted and modiﬁed from Kaczmarek
et al. (Kaczmarek et al., 1991)
Receptor

Class,
type

Receptive ﬁeld(mm2) (Median)

Skin
type

Frequency range(most sensitive)

Threshold skin deform
on hand (median)Probable sensory
correlate

Receptors/cm2
ﬁngertip (palm)

Pacinian corpuscle
Meissner’s corpuscle
Merkel’s cells
Rufﬁni corpuscle
Hair follicle receptor

FA, II
FA, I
SA, I
SA, IIFA

10–1000 (101)
1–100 (12.6)
2–100 (11.0)
10–500 (59)
–

G, H
G
G
C
H

40–800 Hz (200–300 Hz)
10–200 Hz (20–40 Hz)
0.4–100 Hz (7 Hz)
7 Hz?

3–20 lm (9.2 lm)
4–500 lm (13.8 lm)
7–500 lm (56.5 lm)
40–1500 lm (33 lm)–

Vibration tickle
Touch tickle motion Vibr ﬂutter tap
Edge pressure
Stretch shear tension
Touch vibration proximity

21 (9)
140 (25)70 (8)
9 (15)a–FA, fast acting; SA, slow acting; G, glabrous; H, hairy; C, connective tissue.
aKaczmarek et al. originall cited ﬁgures of 9 and 15 cm2 for the density of Rufﬁni corpuscles in the ﬁngerpad and palm, but later work by Paré et al. (2003), Paré et al. (2002)
indicates that Rufﬁni cells are missing from the ﬁnger pads, and insteadare conﬁned to connective tissues, such as the collagen bundles at the base of the ﬁngernails.

Please cite this article in press as: Visell, Y., Tactile sensory substitution: Models for enaction in HCI, Interact. Comput. (2008),
doi:10.1016/j.intcom.2008.08.004

ARTICLE IN PRESS
Y. Visell / Interacting with Computers xxx (2008) xxx–xxx

point threshold, which is the minimum distance between two
point-like indentations applied to the skin below which only asingle point of contact is discerned. This value varies from
2.5 mm in the ﬁngers, up to as much as 50 mm for other body regions (Fig. 3). More recent authors have focused on inadequacies of
the two-point threshold as a measure of tactile acuity (Craig andJohnson, 2000). Much ﬁner features on surfaces can be detected
through dynamic touch, as in the study of LaMotte and Whitehouse mentioned above (LaMotte and Whitehouse, 1986). Tactile
acuity exhibits high inter-individual differences (Craig and Johnson, 2000), and depends very much on the nature of the stimulusthat is used (see the next section for a typology of tactile display
methods), and properties of the stimulus, such as frequency, duration, and amplitude (Cholewiak and Collins, 2000). For example,
Hayashi and Nijouji reported measurements of two-point thresholds due to electrode stimulation of the ﬁngertip at various
frequencies (Hayashi, 2004). A highly salient review of tactile display by Jones and Sarter summarizes the available psychophysicaldata on tactile sensitivity to vibrotactile stimulation on various
body parts (Jones and Sarter, 2008).
Psychophysical thresholds for a particular type of stimulation
indicate the minimum noticable intensity of stimulation Imin , withstimulus properties (other than amplitude) held constant. Maximum stimulus instensity Imax is typically taken to be the thresholdfor pain perception. These values are signiﬁcant for display engineering, as they determine the maximum dynamic range R attainable by stimulation with a particular stimulus type, which may be
expressed in decibels viaRðdBÞ ¼ 10log10 ðImax =Imin Þ

ð1ÞIn general, the maximum dynamic range will be a function of
stimulus type, frequency of stimulation, location on the body, etc.
Sensory transduction is not a linear, temporally or spatiallyindependent process, and complex sensory phenomena result from

Fingers
Thumb
Forearm
Upper Arm
ShoulderForehead
Cheek
Nose
Upper Lip
Breast
Back

Belly

Thigh

Calf

Sole
Hallux0

510

15 20 25 30 35
Mean Threshold (mm)

40

45

50

Fig. 3. Average two-point tactile discrimination thresholds for various bodily
regions (adapted from Kandel et al. Kandel et al., 2000).

5

this. Processes such as temporal and spatial integration create
dependencies between the inﬂuence of stimuli located near eachother in time or spatial position. Locally, sensory thresholds rise
under sustained stimulation. Among other things, this makes it
possible to ignore low-level tactile stimulation, such as thesensation of an object held against the skin. Adaptation is important to account for in tactile interaction design, and for sensory
substitution, because such interventions may involve a continuous
coupling between the user of the system and the stimulating
device over an extended period of time. Tactile sensory thresholds
rise by a few dB in response to only a few seconds of sustained
vibrotactile stimulation, and do not attain a maximum until
approximately 25 min of stimulation have passed. Full recovery
requires on the order of 2 min (Kaczmarek et al., 1991).
As is familiar from everyday experience, qualitative tactile
sensations are wide-ranging and heterogeneous. They include
pressure, texture, puncture, thermal properties, softness, wetness,friction (slip, adhesion, microfailures), dynamic events (contact,release), pain, object features like shape, edges, embossings,
recessed features, and vibrotactile sensations, such as tickling, itch,
vibration, and buzz (Hayward et al., 2004). Many of thesesensations can be associated with different types of tactile
stimulation, several of which are reviewed in the next section.
2.3. Tactile display methods
It is necessary to distinguish tactile display methods from thesensations that they produce and the actuators required to produce
them. Such display methods can be primarily characterized by theformat in which energy is transmitted to the skin, and thereby to
the tactile receptors in it. A brief and non-exhaustive summary ofsuch methods follows.
Low frequency, low amplitude mechanical deformation
Bumps or objects are raised against the skin or other bodypart, or are used to render a relief that can be explored bytouch. One can distinguish between continuous contact withan object, and ‘‘make-and-break” contact, in which an object
is brought in and out of contact with the body part. The skinpossesses an especially high sensitivity to the latter(Kaczmarek et al., 1991).
Vibrotactile stimulation
Consists of objects vibrating against the skin or other bodysurface (for example, the teeth). Typical frequencies arechosen to maximize Pacinian FA II receptor sensitivity(highest near 250 Hz). Vibrations may be effectivelytransmitted through an air gap, again due to the highmake-and-break contact sensitivity. This effect has been
exploited through vibrotactile pin array based devices, suchas the Optacon (described below).
Electrotactile stimulation
Currents are passed through the skin via a source. These
currents excite the afferent nerves directly rather than thetactile receptors themselves. Current may be supplied byelectrodes of different types, or by ﬁne wires inserted into
the skin. Different afferent types can be excited differentiallythrough the design of the drive signal and electrical contacts.
Force feedback displays
Force feedback displays are by nature meant to accessprimarily the kinesthetic haptic channel. However, they interact with the cutaneous tactile sense when friction phenomena,
vibration, or contact transients are inevitably generated.
Thermal displays
Heat is directed toward or away from the skin. The mechanisms of heat transfer can include contact conduction via a
medium, convection, or radiation. Ultimately, the method

Please cite this article in press as: Visell, Y., Tactile sensory substitution: Models for enaction in HCI, Interact. Comput. (2008),
doi:10.1016/j.intcom.2008.08.004

ARTICLE IN PRESS
6

Y. Visell / Interacting with Computers xxx (2008) xxx–xxx

of transfer to heat-sensitive receptors in the body is conduction through bodily tissues. Thermal displays may be readily
implemented using Peltier cells, which can be electronicallycontrolled to pump heat toward or away from an area of skinthat is in contact with it.
Air or liquid jets or currents
Such currents can stimulate SA mechanoreceptors by
supplying forces, FA mechanoreceptors by vibrating skin (Collins, 1970), or hair follicle receptors by moving hairs. They
may supply a regularly varying or dynamic, noisy stimulus.
Thermal effects can also come into play through heat transfer
mechanisms noted above. Focused ultrasonic vibrations in air,modulated at lower frequencies, have been used for spatiallylocalized in-air tactile display (Iwamoto et al., 2001).
In order to implement any of these methods for tactile display, amore detailed accounting is needed of various factors, foremost
among which is the availability and suitability of different actuator
technologies. Moreover, higher-level properties of and structure in
the stimulus signals must be considered, such as:– The total number of actuation sites on the body, and thedensity of sites per unit area. The density may be weighed
relative to spatial tactile discrimination thresholds of thekind described above.
– The size, location on the body, spatial layout and other features of the area to be actuated.
– Temporal qualities of the stimulus, such as its frequency,
and pattern in time. The temporal dependence of tactile
thresholds provide highly salient design considerations (display gain may need to adapt over time to achieve a constantpercept).
– The dynamic range and absolute maximum amplitude of thestimulus. This may be considered relative to the amplitude
threshold and saturation amplitude of the receptors.
– Frequency bandwidth and dynamic range of the device.
Again, these may be measured relative to receptor
characteristics.
– Local features of the interface with the skin at the site ofactuation. Examples include: moistness of the skin, skin biomechanics, such as stretching and compliance, any oils on
the surface of the skin, etc.
– Constancy and reliability of the display; drift.
– Stability of the display in interaction (mainly relevant forforce feedback displays).
– Device accuracy and precision (relative to those of the tactile
channels).
– Safety issues: risks of pain and damage to the user; longterm ergonomics.
– Size, weight, power requirements, portability, anddurability.
Another key detail concerns whether the display targets what issometimes called passive touch, in which case the transducer is applied to a ﬁxed location on the skin, or whether it involves active
touch, in which the area of the display contacted by the skinchanges dynamically as the former is explored.
2.3.1. Vibrotactile stimulation
Vibrotactile stimulation may be the most popular method fortactile display. It involves an object or objects that are made to vibrate against the skin or other body surface for the display of information. The most common method of information coding by asingle actuator is amplitude modulation. If it is desired to transmit
a signal uðtÞ, then the actuator is typically driven by a signal yðtÞgiven byyðtÞ ¼ ½A þ uðtÞ sinð2pfc tÞ;

f c  250 Hz

ð2Þ

As noted above, the carrier frequency is most commonly
selected in this way to best address the FA II (Pacinian) mechanoreceptors in the skin. The resulting stimulus elicits a sensation thatis felt as a bump, a vibration (possibly one possessing the signatureof a physical process, such as rolling or sliding), a texture or
perhaps a material quality.
Cross-modal sensations and illusions can also be elicitedthrough vibrotactile stimulation. An appropriately chosen stimulus, applied to the muscle and tendonal mechanoreceptors, canmake the limbs seem to move in a manner that can be controlled.
Roll and Gilhodes (Roll and Gilhodes, 1995) studied the organization of illusory vibrotactile proprioceptive sensation in movement
trajectory perception. Their system has elicited the feeling of selfgenerated drawing movements in subjects by applying vibrotactile
patterns to their wrist muscles. Such illusions have been exploitedfor the purpose of generating whole arm and upper body movement illusions via vibrational feedback in subsequent work by
the same authors.
2.3.2. Electrotactile stimulation
Electrotactile information displays have been investigated atleast since the early 1970s (Triggs et al., 1973). Electrotactile stimulation involves a current passed through the skin via a current
source. The current elicits a sensation by exciting afferent nerves
directly. Secondary electrotactile effects can result in cases inwhich a small current modiﬁes the frictional properties of a surface. Electrotactile stimulation can evoke a wide range of sensations, including, itching, tingling, vibration, buzzing, touch,pressure, pinch, or pain. Currents may be supplied by an electrodeon the skin or by a ﬁne wire or other structure inserted into or beneath the skin (Kaczmarek et al., 1991). In sensory studies, single
afferent ﬁbers are stimulated with microelectrodes to reveal thesensations associated with activation of different ﬁber types (Torebjork et al., 1984, as related in Kaczmarek et al., 1991). Newer
methods of electrotactile stimulation attempt to differentiallyevoke desired sensations by controlling the mode of stimulation.
Several difﬁculties must be overcome for electrotactile stimulation to be practical (Kaczmarek et al., 1991). First, the electrodetype and material are signiﬁcant, as are the skin-electrode coupling, including contact force, skin location, skin hydration and oiliness. Hairy and glabrous skin pose challenges due to the fact thatthe skin conductivity can change rapidly as pores open or close or
as sweat accumulates on or under the top layers of the skin. In theabsence of careful current control, such changes can lead to painfulshocks. Mucous skin like that of the mouth has beneﬁts for electrotactile stimulation, as discussed below, because tactile receptors
are closer to the surface and the tissue is kept moist.
2.3.3. Selective electrostimulation of tactile mechanoreceptors
Work by Tachi and his colleagues (Kajimoto et al., 2004) has
focused on the development of electrotactile displays for theglabrous skin that are capable of differentially activating mechanoreceptors – a phenomenon they have described as ‘‘tactile color”(see Fig. 4. Their work is based on a study of the electrical physics
of the skin in relation to the geometry and composition of afferent
neurons in the skin. The main points are as follows. The axons of
the primary mechanoreceptors turn out to be of larger diameter
than others present in the skin (i.e. those corresponding to thenociceptors, themoreceptors, and so on). This translates into alower activation potential for the mechanoreceptors, so that they
are activated ﬁrst as current is increased. The Meissner corpuscles(FA I) possess vertically oriented afferent nerves, while that of the
Merkel receptors (SA I) is horizontally oriented. When stimulated
by a coaxial electrode pair, the vertically oriented afferent of the

Please cite this article in press as: Visell, Y., Tactile sensory substitution: Models for enaction in HCI, Interact. Comput. (2008),
doi:10.1016/j.intcom.2008.08.004

ARTICLE IN PRESS
Y. Visell / Interacting with Computers xxx (2008) xxx–xxx

7

Skin surface
Merkel cell(pressure)

Meissner corpuscle(low-frequency vibration)

Pacinian corpuscle(high-frequency vibration)

Fig. 4. The Smart Touch project the Tachi Laboratory, The University of Tokyo (Kajimoto et al., 2003, 2004) aims at enabling the composition of tactile sensations by means of
selective electrical stimulation of tactile mechanoreceptors (image courtesy of Tachi Laboratory, University of Tokyo).

Meissner corpuscle is more susceptible to activation when the central electrode of the pair is a cathode. When stimulation is suppliedin this way, a vibratory sensation is said to be felt. Conversely, thehorizontally oriented afferent of the Merkel corpuscle is more susceptible to activation when the central electrode is an anode, andconsequently under this mode of stimulation, pressure sensations
are said to be felt.
The Pacinian corpuscles lie deeper within the skin, and a larger
electrode current is required to activate them for this reason. Further work by Tachi and his colleagues has aimed at selectively
stimulating these receptors through the use of arrays of electrodes.
2.3.4. The Tongue Display Unit system
The Tongue Display Unit (TDU) is an electrotactile sensory substitution device developed by Bach-y-Rita and his colleagues(Bach-y-Rita et al., 1998). The device (Fig. 5) consists of an input
sensor coupled to a two-dimensional electrotactile array placedon the tongue. A signiﬁcant advantage of tongue-based electrotactile display is that the tongue provides a surface that is consistently
moist with saliva, ensuring good electrical contact. In addition, thesensory receptors lie closer to the surface of the tongue than they
do in most of the body, including the ﬁngers, so lower voltage andcurrent can be used for stimulation – about 5–15 V and 0.4–2.0 mA(Bach-y-Rita et al., 1998). Compared with the glabrous or hairyskin, electrotactile stimulation of the tongue is safer (due to thelower power), poses fewer engineering problems and is more consistent (due to the favorable electrical conditions).

Camera144-channel ribbon cableVideo cable

Later versions of the TDU have been designed to be ﬁtted to acustom denture which is worn in the mouth, including wirelesstransmission from a portable base unit. The device has been usedfor visual to tactile substitution (Fig. 5) and for vestibular to tactile
substitution (Section 3 below), in which case it is coupled, respectively, to a video camera or to an inertial sensing device.

3. Electronic sensing for sensory substitution
Electronic sensors of some form are required to capture theinformation associated with the substituted modality that is tobe displayed through touch. Table 2 provides an incomplete listof sensing methods for sensory substitution. Methods can be characterized according to the sense being substituted, the main sensor
device types that are used, the type of environmental information
the sensor captures, and the range of applications for which they
are useful.
A number of other features of the sensing apparatus and its
integration in the substitution device must also be addressed.
These include:– The range and ﬁeld of view to be displayed, which has beenfound to affect recognition of familiar symbols in ways thatare difﬁcult to predict (Wall and Brewster, 2006).
– The sensor location, whether on the body or mounted elsewhere, which determines the perceptual coordinate inversion that must be solved if the information is to be used insome motor task.
– The mode of user control over the sensor, typically via motor
behavior of some form.
– The spatial and temporal resolution, bandwidth, and othercharacteristics of the sensor.
– The relation of the sensing method to the context and task –for example, the environment of the home vs an assemblyline production.

TDU

Such questions tend to depend on the type of sensing method
used, and choices may be inﬂuenced by the application, display
method and coupling possibilities.
3.1. Example: electrotactile displays for vestibular disorders andproprioception
Fig. 5. The Tongue Display Unit (TDU) of Bach-y-Rita and his colleagues (Bach-yRita and Kercel, 2003), showing a conﬁguration designed for visual to tactile
substitution Right: TDU display of Vuillerme et al. (2007a).

The Tongue Display Unit (TDU) of Bach-y-Rita and hiscolleagues (Bach-y-Rita, 2004) has been used as a balance
prosthesis in work by Vuillerme et al. (Vuillerme et al., 2007a)

Please cite this article in press as: Visell, Y., Tactile sensory substitution: Models for enaction in HCI, Interact. Comput. (2008),
doi:10.1016/j.intcom.2008.08.004

ARTICLE IN PRESS
8

Y. Visell / Interacting with Computers xxx (2008) xxx–xxx(Fig. 6). Their system consists of a pair of force sensing shoe solesthat are used to determine the position of the center of pressure
applied by the individual to the ground. This position provides anindication of the direction and amount of postural sway (butfurnishes limited information about dynamic sway in the upperbody). The sway is displayed through the TDU by means of adiscrete signal indicating when the center of pressure exceeds abound in the medial–lateral or anterior–posterior directions. As
is the case in many biofeedback systems designed for rehabilitation, the information supplied by the feedback channel is reduced
from a complex set of variables (in this case, postural) to a small
number of discrete indicators. In separate work (Vuillerme et al.,2006), Vuillerme et al. applied the same TDU device to the displayof joint angle position information in a task requiring the matchingof ankle joint angle conﬁguration by healthy subjects. The display
consisted of a six-by-six electrotactile array applied to the tongue.
The encoding consisted of discrete cues to indicate when the subject has positioned the angle in an overly plantarﬂexed position
relative to the reference (opposite) ankle, or in an insufﬁciently
plantarﬂexed position. The cues themselves consist of stimulating
the tongue at 12 out of the 36 sites, either in the posterior or anterior end of the electrotactile array. Feedback was found to aid theability of subjects to correctly match the ankle joint angle in thistask. No comparison was made between the discrete electrotactile
cues that are provided and any other form of feedback. The authors
suggest that some speciﬁc mechanism may be at work, whereby
the CNS combines the electrotactile information with proprioception to complete the task. It is difﬁcult to evaluate such a claim,due, not least, to the rather simple nature of the stimulus as an error indicator. One could also argue that the correct behavioral re-sponse can be learned via classical conditioning, which may notdepend on any special sensory integration mechanism.
4. Coupling sensing to actuation
In addition to display and sensing devices, a coupling method
mapping the second to the ﬁrst is required to complete the interaction loop of the substitution apparatus, as shown in Fig. 7. The
coupling maps information xt transduced by the sensors into signals yt suitable for driving the actuators of the display. In interesting examples, it need not be a static (memoryless) function, nor alinear map.
Requirements for coupling vary considerably by application.
Some considerations include the domain of the electronicallytransduced sensory signals, and that of the target actuator signals.
Properties of intrinsic human sensory pathways and information
processing capabilities associated to the display modality are signiﬁcant. Finally, human sensory information processing capabilities nominally associated with the electronic sensing modality(for example, human vision, in the case of video-based sensing)may be important, if those sensory processing pathways are conjectured to be able to be recruited for processing the displayed
information.
Sensor and actuator conﬁgurations can be characterized, in part,by their spatial and temporal domains. Examples include one- or
two- dimensional spatial arrays (also dependent on time), as in video data; non-spatially arranged vectors, such as those from kinematic or force data. A single point transducer may vary in a more orless complex way over time. This variation may be proﬁtably
thought of in terms of its time-varying spectral contents, as for

Table 2
Selected sensing methods used in tactile sensory substitution
Modality

Sensor type

Information captured

Application/examples

VisionAuditionVideo cameras or light sensors
Microphones

Visual appearance of environment
Acoustics (ambient, interactions)Distance sensing (sonar)

Ultrasonic; IR ranger; other distancesensor; arrays
Piezoelectric elements or other vibrationtransducers
Strain gauges, other force sensors
Position or rotation encoders; inertial
sensors; motion capture
Inertial sensing; compass; motion capture

Distance to nearest object structure ofenvironment
Contact interactions, vibrations

TVSS Epstein et al. (1986)
Hearing aid/Tactaid Reed and Delhorne(1995)
Mowat electronic travel aid

Vibration, friction (touch)Force (cutaneous)
Proprioception
Vestibular

Force
Sense position of & forces in the body;
joint position, posture
Sense balance related information

Teleoperation display Massimino (1991)
Haptic teleoperation (Debus et al. (2001)
Biofeedback – e.g. TIKL Lieberman andBreazeal (2007)
Vestibular aid TDU (Bach-y-Rita et al.)Fig. 6. The TDU-based balance prosthesis of Vuillerme et al. (Vuillerme et al., 2007a). Left: The pressure mapping and corresponding stimulation pattern for the TDU given theshown body posture (Vuillerme et al., 2007b). Right: The wireless version of the TDU, a 6  6 electrotactile matrix ﬁtted to a custom denture with an embedded radiofrequency receiver in the mouth of the subject.

Please cite this article in press as: Visell, Y., Tactile sensory substitution: Models for enaction in HCI, Interact. Comput. (2008),
doi:10.1016/j.intcom.2008.08.004

ARTICLE IN PRESS
Y. Visell / Interacting with Computers xxx (2008) xxx–xxx

Movementof device

Sensors(Modality A)

Body

x(t)

Coupling
Devicey(t)Actuators(Modality B)

Environment

Fig. 7. The coupling component in a sensory substitution system, shaded in grey.

example with audio data captured from a microphone, or it might
be better characterized according to its temporal structure (e.g.
roughness or rhythmic pattern).
4.1. Information coding in sensory substitution
On the surface, information coding refers to way in which information that is acquired through the sensing apparatus is presented
to the display. This presentation should be performed in such a
way that the displayed information is useful for the human sensory
apparatus that is involved. At the most basic level the relevant sensory and psychophysical properties surveyed in Section 2 must be
respected. Equally importantly, the information should be displayedso as to be useful for processing in the central nervous system. Suchtactile information processing capacities have been the subject ofconsiderable attention in the literature on tactile display and perception (Gallace et al., 2007). Due to the cross-modal nature of sensory
substitution devices, this raises many complex questions. For example, it is not a priori clear that human sensory processing resources
for tactile information would have any success at extracting information from video data, given the relatively specialized neural pathways devoted to visual processing. As noted below, however, somehave seen promise in results of several studies on cross-modal processing in the brain.
4.1.1. Tactile display encodings
Consideration as to the manner in which the input signal will be
coded for presentation by the display device is needed. The simplest encodings employ a one to one mapping from the domainof the sensor to the actuator array, when the two are homeomorphic. If there is a mismatch in the number of elements in the displays, simple resampling methods may be used.
A mapping of temporal or frequency features to spatial features
is sometimes employed – for example, in auditory to tactile substitution. The Tactaid VII device (Section 5.1.0.2) is such an example.
In such cases, the level of activation of each actuator in a linear array can be chosen to correspond to the amplitude of the sensed signal in a given frequency band.
Position coding is frequently used to display a continuous, scalar value by means of a one-dimensional array; typical encodings
include ‘‘1-hot”, where a single actuator element is excited along
the array at a position corresponding to the value, or thermometerencoding, where all actuators up to the given position are turned
on. Similar two-dimensional positional encodings have also founduse. The vestibular substitution of Vuillerme et al. is a simple
example (Vuillerme et al., 2007a).
Van Erp and his colleagues have made extensive use of vibrotactile displays worn on the torso as directional or position cues. The
authors have conducted psychophysical studies of the localizationof vibrotactile stimuli on the torso, the dependence of localization
on stimulus parameters, and the indication of spatial directions

9

using these stimuli (van Erp, 2005,). Applications from their work
have included the display of orientation information via a wearable
tactile vest (Fig. 8) for astronauts in zero gravity (van Erp and vanVeen,2003, 2006), and the display of directional information for pedestrian wayﬁnding (van Erp et al., 2005) and in-vehicle navigation(van Erp, 2004). Others have similarly used position encoding
within a vibrotactile array to indicate a directional vector, withor without magnitude display. Hein and Brell (Hein, 2007) presented a vibrotactile glove with several embedded tactors for the
display of positioning information to a surgeon during computerassisted surgery.
A tactile illusion called sensory saltation has been used by Tan
and others to display a position on the skin via vibrotactile feedback (Tan and Pentland, 1997). The illusion creates the sensationthat the point of actuation is located within an area of ﬁner spatial
resolution than the physical actuator array layout.
Ecological encodings can be used to exploit innate human abilities to perceive dynamical properties of physical systems fromtheir vibrational signatures, such as the sound or feel of a rollingball (Yao and Hayward, 2006; Rath and Rocchesso, 2005).
4.2. Evaluating coupling methods
The choice that is involved in constructing a coupling fromsensing to actuation is frequently overlooked in the literature. In
some cases, it is possible to independently evaluate the coupling
method, comparing candidate choices of coupling map. Perceptual
evaluations can be designed to measure information transfer to theusers through the channel, based on users’ ability to identify thestimuli with which they are presented (Tan et al., 1999). Or evaluations be task-based, asking users to complete tracking (van Erp
and Verschoor, 2004) or control (Kadkade et al., 2003) tasks. The
degree to which users of such a device may be able to adapt to aparticular choice of coupling map over time remains somewhat
underexplored.
Kadkade and his colleagues (Kadkade et al., 2003) evaluateddifferent methods of vibrotactile coding using an unstable,
one-dimensional manual control task. Their display consisted of

Fig. 8. A vibrotactile vest, presented in (van Erp and van Veen, 2003), used for the
display of orientation information for astronauts in zero gravity (ﬁgure reproduced
from van Erp and van Veen, 2003). Direction of a reference position is displayed by
activating a single tactor of the vest (an example of ‘‘1-hot” encoding).

Please cite this article in press as: Visell, Y., Tactile sensory substitution: Models for enaction in HCI, Interact. Comput. (2008),
doi:10.1016/j.intcom.2008.08.004

ARTICLE IN PRESS
10

Y. Visell / Interacting with Computers xxx (2008) xxx–xxx

Fig. 9. Pictorial collage of sensory aid devices. Left to right: The Braille alphabet, the long white cane, the VTPlayer mouse (a mouse with integrated tactile pin array), an
electronic Braille reader, and the Optacon.

a linear array of tactile vibrators applied to the back. Several
candidate codings were evaluated, including 1-hot position
encoding, with a single active tactile vibrator, or tactor, providing
position feedback. The authors also evaluated a spatially discrete
display consisting of one tactor on the right and another on the left
side, excited at intervals. In this conﬁguration, the rate of excitation encoding the amount of error, and the side encoded direction.
The position based encoding scheme was found to provide better
performance on the manual control task.

5. Perspectives and conceptual models
Accounts of the creation, engineering and assessment of sensory
substitution devices throughout the past century have beenstrongly inﬂuenced by the prior conceptions that have motivatedgroups researching the devices. Conceptual viewpoints have variously been guided by ideas about the creation of sensory prosthetics, the provision of unseen environmental information, studies of
the adaptability of human perception and cross-modality, andlong-standing questions about the phenomenology of perception.
These notions are or broader signiﬁcance for enaction and enactive
interfaces, and have frequently arisen in discourses on these subjects. To do justice to all of the ideas involved would exceed thescope of this review. The theoretical issues related to perception,
while fundamental to the design and understanding of tactile sensory substitution systems, are both wide in scope and, at times,
controversial.2 Nonetheless, this section collects some of the relevant ideas.
5.1. Sensory assistancetranslate the patterns of raised dots into characters (Foulke,1982). It has also been the subject of a number of studies on the
difference in tactile performance between sighted and blind
people.
Standing in contrast with the symbolic nature of these communication aids is the long cane, a travel aid that is now commonlyused by visually impaired pedestrians. The cane conveys information about the structure of the environment in a continuous way,
through the haptic transmission of contact between the far endof the stick and ground, and through the sound that is produced
through contact such as tapping. The long cane is a commonly cited example of the phenomenon of distal perception (Merleau-Ponty, 1945; Kreuger, 1970; Gibson, 1962), which refers to the fact
that, that when employing it, users do not attend to the proximal
haptic signal at the interface of their hand with the cane. Rather
their tactile perception is focused upon the distal stimulus, at thelocus of contact between the cane and the ground. Fig. 9 shows
several examples of sensory aids (see Fig. 10)
5.1.0.1. Example: the Optacon
The Optacon is a tactile sensory substitution device developed
by Bliss and his collaborators beginning in the early 1960s. The
Optacon was designed as a reading aid, and later adapted to supply
visual environmental information via touch, through the additionof a lens with a more distant focal plane (Bliss et al., 1970). It possesses a tactile display consisting of a 6  24 pin array designed forthe ﬁnger pad. A version of the Optacon was available for sale froma company called TeleSensory until the device went out of production in the mid 1990s (Wall and Brewster, 2006). The decline in
popularity of the Optacon coincided with a growth in interest byvisually impaired readers in page scanners based on optical character recognition (OCR), and eventually in computer-based screen

As noted in the introduction, sensory substitution has at times
been equated to the provision of devices to aid people who experience sensory deﬁcits such as blindness or deafness. The development of the TVSS and other systems like it were motivated inthis way (Bach-y-Rita and Kercel, 2003). A wide range of sensory
aid devices has been developed to date. Two important categories
are communication aids and travel aids – devices, in other words,
that help people with sensory deﬁcits to understand communication, or that aid perception for locomotion and transit. Amongthe earliest communication aids are Braille, the system of text display for visually impaired people that was developed by Louis
Braille in the 1840s; and sign language, a gesture-based substitution for spoken language for use by deaf people, with its roots in18th century France. Braille and sign language were designed toconvey symbolic information in ways that circumvent the
impaired sensory channel, albeit at different levels of abstraction(Braille’s symbols are characters, while those of sign language are
linguistic elements – words or phrases). Braille reading is a task
requiring a high level of tactile information processing, needed to2
A wide range of opinions on the ideas noted in Section 5.6 are represented in thecommentaries accompanying the seminal 2001 article on sensorimotor behavior invisual consciousness by O’Regan and Noë (O’Regan and Noë, 2001).

Fig. 10. The optacon device is used both for reading printed text and for viewing anenvironment, both through its 6  24 pin tactile display. As noted in the original
caption of this ﬁgure, reproduced from Bach-y-Rita et al. (Bach-y-Rita and Kercel,
2003), the child is able to mimic the hand posture of the teacher using feedback
acquired via the device.

Please cite this article in press as: Visell, Y., Tactile sensory substitution: Models for enaction in HCI, Interact. Comput. (2008),
doi:10.1016/j.intcom.2008.08.004

ARTICLE IN PRESS
Y. Visell / Interacting with Computers xxx (2008) xxx–xxx

Fig. 11. The Tactaid auditory-tactile substitution device.

readers, both of which require much less training to use than the
Optacon. In addition to aiding individuals with visual deﬁcits, theOptacon has been used in fundamental studies on sensory substitution and tactile perception.
5.1.0.2. Example: the Tactaid
The Tactaid VII is an auditory-tactile substitution device thathas been made commercially available to aid hearing impaired
people.3 The device is shown in Fig. 11. It is worn on the forearmor on another body part. Ambient acoustic signals are captured by
the microphone, and their intensity is measured in seven frequency
bands. The resulting amplitudes are mapped to the vibration amplitudes of seven actuators on the device, spaced in a linear array, using
the frequency band coding described above. The device has beenfound to improve speech sentence understanding by about 10% after
training (Reed and Delhorne, 1995).
5.1.1. Sensory substitution as rehabilitation
The term rehabilitation is commonly applied to describe a process whereby a patient afﬂicted with some physical or cognitive
disorder is assisted in recovery. It is interesting that the label is also
applied to sensory substitution, because the sensory deﬁcit is notnormally healed by means of the substitution, but is instead circumvented through the intervention of the device. The Oxford
American Dictionary offers the following deﬁnition.
Rehabilitate (verb)(i) Restore to health or normal life by training and therapy after[. . .] illness.
(ii) Restore (someone) to former privileges or reputation after aperiod of critical or ofﬁcial disfavor.
(iii) Return to its former condition (applied to environmental
state).
In English the word originates in the late 16th century, and refers
to the ‘‘Restoration to former privileges”, such as those conferred bya title. This usage in turn comes from the medieval latin verb habilitare, which means ‘‘To clothe or equip” and ‘‘To make able to”.
The description of tactile sensory substitution as ‘‘rehabilitation”
seems, from the etymological standpoint, apt. First, because thesubstitution confers an ability or privilege (that of sight, for example), and is aimed at a restoration of ability after it has been lost.
Second, because it evokes the idea of a surrogate sensory channel
as re-clothing the individual with the sensory layer that it has lost,a notion that speciﬁcally resonates with the idea of touch.
5.2. TSS as the display of hidden features of an environment
Tactile sensory substitution has been viewed by various
researchers as the attempt to replace sensory deﬁcits through3

It is currently sold through the following website: http://www.tactaid.com.

11

touch. This idea can be extended to include the display of sensory
information associated with an environment (Kaczmarek et al.,
1991; Lenay et al., 1997; Lenay et al., 2003), even if this information was not lost, but is merely hidden, i.e. unavailable to an individual acting in that environment. Such a characterization would
include sensory replacement devices, as well as the provision ofsuper-normal sensory capabilities. The environment about which
the user receives this information could be the real space occupiedby the user, or one in which the user acquires information about adifferent space, such as a virtual environment, a simulation, or thesurroundings of a slave robot controlled via teleoperation in a remote environment. When it is not the user’s immediate environment that is being portrayed, the sensory substitution may
constitute the entire experience of the space, which may be otherwise fully disconnected from the user’s sensory capabilities.
5.2.1. Vibrotactile (VT) feedback in teleoperation or VR
Vibrotactile (VT) feedback has been used in several ways inteleoperation and virtual reality. Information displayed by such asystem may include:– Surface textures and friction with surfaces in the remoteenvironment or simulation (Minsky et al., 1990, Fujimoto
et al., 2004).
– Contact events (Massimino, 1991).
– Shape related features (Cohn et al., 1833; Hasser, 1993;
Kontarinis and Howe, 1995).
– Dynamic physical processes, such as rolling (Yao andHayward, 2006) or breaking.
– Force magnitude and/or direction, speciﬁcally arising fromcontact forces (Massimino, 1991; Debus et al., 2001, 2002).
In addition to fulﬁlling the essential requirement of revealing a
remote environment to an operator, VT sensory feedback offers
several advantages in such settings. It can be implemented in waysthat are cheap, compact, low power, and simple. It is well suited todisplaying information for skilled manipulation and ﬁne motor
control. The ability to perceive elements of a remote environment
can enhance presence, or the sense of being there. It can also improve users’ understanding of the forces involved in interaction,
reducing overexertion in cases in which the preexisting force data
is not clear (for example, during the use of a space suit). Vibrotactile feedback may be used as a substitute for force display, which
can be beneﬁcial because VT feedback does not destabilize interaction loops in the way that force feedback is capable of doing. VT
feedback has also been used in combination with force feedback,where it seems to be most efﬁcient for display of frequency components above the lowest resonant frequency of the force feedback
interface (Kontarinis and Howe, 1995). The added vibrotactile
information may be supplied through the motors of the haptic
interface, or via dedicated vibrotactile actuators.
Evaluations of the effectiveness of augmenting teleoperation or
VR tasks with VT feedback have been decidedly mixed (Browse and
McDonald, 1992). In situations where other modalities provide
clear information about the environment’s state relative to the task(for example, clear visual feedback in the case of an assembly task),
VT feedback does not seem to improve performance (Massimino,
1991; Debus et al., 2001). Common performance measures include
the time to complete a task, the amount of energy required to complete it, the peak force or excess force used in carrying it out. Subjective improvements can be assessed via qualitative tools, such as
the Nasa Task Load Index (Hart and Staveland, 1988).
5.2.1.1. Example. VT augmentation of a haptic manipulandum(Debus et al., 2002) Debus and his colleagues studied VT
enhancement of a PHANTOM haptic interface for the display ofPlease cite this article in press as: Visell, Y., Tactile sensory substitution: Models for enaction in HCI, Interact. Comput. (2008),
doi:10.1016/j.intcom.2008.08.004

ARTICLE IN PRESS
12

Y. Visell / Interacting with Computers xxx (2008) xxx–xxx

forces at the peg-like end effector of a slave robot (Debus et al.,2002). The device they created is a VT enhanced manipulandum
handle that displays force magnitude and direction during interaction with a virtual environment (Fig. 12). The handle contains
four vibrotactile actuators arranged around the circumference ofthe handle, each of which is coupled to the skin via a pin array.
The vibration amplitude of each actuator is designed to be proportional to the component of a force in the plane orthogonal
to the principal axis of the peg at the end of the slave robot.
As suggested above, performance improvements accompanying
this augmentation were limited. In a peg-in-hole task, subjects
tended to use 10% less force for task completion as compared withthe case of visual only feedback. No improvement in mean task completion time was found.
5.3. Cross-modal plasticity
Many authors have discussed cross-modal plasticity in the brain
as an explanatory mechanism for sensory substitution. Plasticity inneuroscience refers to the nervous system’s ability to change structure and function in response to experience or change in operating
conditions (Kolb and Whishaw, 1998). Changes, which may affectperception or other functions, can be continuous (due to modiﬁcations of the environment, developmental experience, or perturbations to the perceptual system (Clifford, 2002; Goldstone, 1998)
or discontinuous (as in the case of brain lesions due to stroke, accidental blindness, etc). Subjected to discontinuous changes, thebrain is capable of both restoration of function in damaged pathways and of reorganization, in which functions associated to one
brain region are instead directed to another (Bach-y-Rita, 2001).
Pascual-Leone et al. argue that plasticity is an intrinsic propertyof the nervous system, part of its normal ongoing state throughout
the life span (Pascual-Leone et al., 2005). They suggest that it is notpossible to understand normal psychological function or disease
without invoking it.
Cross-modal perceptual reorganization has been hypothesized
to be salient to sensory substitution, because it has been supposedthat neural resources associated to the substituted modality can be
recruited to process the same information as displayed through another sensory input. Such a reorganization, if possible, could allow
users of a sensory substitution device to take advantage of existingneural circuitry associated to the substituted modality. For example, it has been suggested that areas of the brain normally used
for processing visual information could be employed to process
information displayed to the skin via a tactile vision substitution
device (Bach-y-Rita and Kercel, 2003,).
Bavelier and Neville review recent knowledge on cross-modal
plasticity, citing evidence from healthy animals and people, thosewith hereditary sensory deﬁcits, and those that have experienced
deﬁcits due to lesions of some form (Bavelier and Nevile, 2002). Théoret, Merabet, and Pascual-Leone discuss evidence of neural-reorganization in adjustment to blindness. They cite ﬁndings ofenlargement in the somatosensory cortical representation of bodyparts involved in increased tactile experience (such as Braille reading (Pascual-Leone and Torres, 1993; Hamilton and Pacual-Leone,1998), and recruitment of visual areas of the brain for tactile andauditory processing in early blind individuals (Théoret et al., 2004;
Pascual-Leone, 2001). Théoret et al. note that the correspondence
between behavioral studies and evidence of neural reorganization
within the brain have not generally been clear. For example, while
there have been conﬂicting results on changes in performance within the remaining senses after blindness, evidence of neural reorganization has been less ambiguous (e.g., Amedi et al., 2007). However,
the relevance of the latter to perception and behavior is more difﬁcult to assess. The idea of plasticity, as introduced by William Jamesin the 1890s, was advocated later by Ramón-y-Cajal as an explanatory mechanism for behavioral modiﬁability in terms of changes inbrain anatomy. Yet as various authors have noted (Pascual-Leoneet al., 2005; Clifford, 2002), the mapping between the two is far frombeing one to one. This ambiguity complicates the picture of neuralplasticity as a mechanism in sensory substitution. Consequently, itremains difﬁcult to draw ﬁrm conclusions about sensory substitution from the many recent brain imaging studies that have found evidence of cross-modality. Poirier et al. provide some further analysis
and evidence in support of neural cross-modal plasticity as a mechanism speciﬁcally enabling sensory substitution (Poiriera et al.,2007).
5.3.1. Corrolaries to plasticity
The conceptual implications of plasticity in sensory substitution
are intriguing. First, plasticity highlights the way in whichperception may adapt to utilize different information sources,and by consequence, perhaps, different display modalities. Bachy-Rita argues, ‘‘we see with the brain” (Bach-y-Rita, 2003), not withthe sensory organ. In studies with the TVSS, researchers noted that
if the display is moved to a different area of skin – say, from theskin on the back to that on the forehead – functional use of the device is recovered almost immediately relative to the time that was
required to ﬁrst learn to use it (Bach-y-Rita and Kercel, 2003). Similar rapid adaptation has been reported if the sensor is moved fromone location to another – for example, if the video camera is moved
from the head to the hand. Perceptual plasticity has been proposed
as an exploration for this form of site-independence.
Many open issues exist surrounding the degree to which neuralplasticity may be involved in the use of sensory substitution devices, or in substitution through intrinsic perceptual channels.

Fig. 12. The VT enhanced PHANTOM manipulandum of Debus and his colleagues (Debus et al., 2004). Left: A diagram of the master (PHANTOM) – slave (robot) system
conﬁguration. Center: The VT augmented manipulandum handle. Right: The actuated handle, as seen in 2D projection.

Please cite this article in press as: Visell, Y., Tactile sensory substitution: Models for enaction in HCI, Interact. Comput. (2008),
doi:10.1016/j.intcom.2008.08.004

ARTICLE IN PRESS
Y. Visell / Interacting with Computers xxx (2008) xxx–xxx

13

Fig. 13. The minimal sensory substitution experiment of Lenay and his colleagues (ﬁgures reproduced from (Lenay et al., 1997).

Among the earliest to pose the question explicitly was William
Molyneux (whose spouse was blind). Posing what is now knownas ‘‘Molyneux’s Question”, he wrote to John Locke in 1688, asking
whether a person blind from birth who had learned to distinguishby touch between a sphere and cube would, upon suddenly gaining
sight, be able to distinguish the two forms visually (recounted by
Morgan (Morgan, 1977) and later discussed by Bach-y-Rita andKercel (Bach-y-Rita and Kercel, 2003). Molyneaux’s question
evokes the idea of cross-modal transfer of knowledge, the questionof equivalence between two senses, and the role of behavior inreﬂecting perception (Degenaar and Lokhorst, 2005). Morgan
boldly asserts that ‘‘There is not the slightest reason to think that
if two sensory messages give the animal exactly the same information, and lead to exactly the same behavior, they will be perceived
differently – even if they come over completely different pathways” (Morgan, 1977).
5.4. Distal perception
Distal perception refers to the notion that stimuli received on
the skin by means of the TSS device may become associated witha distant location in the environment, or even perceived as having
an origin there. The location would correspond to the end of a stick(in the case of exploration of the environment with a cane) or apoint attended to by a camera in a TVSS system. It can be deduced
according to the laws that govern the change in stimulation received as the sensory device’s orientation and position are changed. Several authors have emphasized the notion that sensations
can be externalized (attributed to environmental sources outside
the self) only when this afferent stimulation depends on ones actions (Loomis, 1992; White, 1970; Epstein et al., 1986).4 In tactile
perception, Gibson, among others, noted the difference between
being touched, as a sensation arising from the stimulation of ones
skin, as compared to actively touching, in which the sensation isassociated to properties of the external object (Gibson, 1962; Kreuger, 1982).
The potential for distal attribution in tactile sensory substitution offers another perspective on studies that have found stimulation site invariance (cited in the preceding Section (Bach-y-Rita
and Kercel, 2003). The idea is that the source of the stimulation
comes to be attributed not to something on the skin, but to thelocation in space that is consistent with the stimulation that is received as the orientation and position of the sensor vary. Environmental attribution, if accurate, suggests that the experienced
phenomenon may be largely invariant upon displacement of thedisplay to a different area of the body because the stimulus refers
to a determinate location in the surrounding space.

4

When afference is independent of ones actions, it has the same character as suchinternal experiences as hearing ones heartbeat, or feeling a tingling in the extremities.

5.5. Sensorimotor behavior and externalization
Some authors have gone further, and emphasized the role ofsensorimotor invariants as a fundamental feature of perception,
to be studied through TSS devices (Lenay et al., 1997, 2001,
2003; Auvray et al., 2005) and simulations (Philipona et al., 2003,
2005). Lenay and his colleagues conducted a set of experiments
to investigate the role of sensorimotor behavior in externalization(Lenay et al., 1997). They created a device worn on the ﬁnger consisting of a single photoreceptor with an angle of sensitivity of
approximately 30° coupled to a single vibrotactile actuator witha simple on/off capability. The environment used in their experiments consisted of a single point light source in an otherwise dark
space. In the absence of motor control over the device, its user cannot ‘‘see”, because all that can be inferred is whether the lightsource stands in front of it or not. Granted control, over the orientation of the sensor (via the direction the ﬁnger points to) and its
position, a user of the device can infer the direction to the lightsource, and by changing its position, can infer the position, throughparallax. One way to explain this is that through interaction, theuser infers the invariant relation:

L ¼ bðsin a  cos a tanða þ bÞÞð3Þ

Here, L is the distance to the light source, b is the length of thearm, a and b are the arm and wrist angles, respectively, as depicted
in Fig. 13. The stimulus’ origin can be readily distinguished using
this formula.
Current commercially available electronic travel aids for visually impaired people function in a fashion that is somewhat analogous to the device of Lenay et al. Handheld sonar devices such as
the Miniguide5 or the Mowat6 vibrate to indicate the distance tothe nearest object in their line of sight. The Miniguide costs approximately US$380. Several related electronic travel aids are reviewed
in a survey on tactile sensory aids by Levesque (Lévesque, 2005).
5.5.1. Cognitive information in externalization
While many studies on sensory substitution have found evidence that subjects were able to externalize the sensations theyfelt, in most of these experiments, including that of Lenay et al.,subjects are made aware of the fact that the sensory objects theyexperience are the result of physical objects in their surroundings(the light source, in the case of (Lenay et al., 1997). Comparatively
few studies have assessed externalization in the absence of suchcognitive information (Auvray, 2004). Epstein et al. designed avibrotactile sensory substitution system based on the stimulationof the index ﬁngertips of blindfolded individuals, who were allowed to experience transformations of the stimulation that were
5
The Miniguide ultrasonic mobility aid. http://www.gdp-research.com.au/
ultra.htm.
6
The Mowat sensor. http://www.as.wvu.edu/scidis/terms/mowat.html.

Please cite this article in press as: Visell, Y., Tactile sensory substitution: Models for enaction in HCI, Interact. Comput. (2008),
doi:10.1016/j.intcom.2008.08.004

ARTICLE IN PRESS
14

Y. Visell / Interacting with Computers xxx (2008) xxx–xxx

correlated with self-movement (Epstein et al., 1986). These subjects were found to have developed an awareness of the sensorimotor contingencies between their actions and the tactile stimuli
they received. However, subjects were reported largely to havefailed to externalize these stimuli.
5.6. Sensory substitution as sensorimotor augmentation
As illustrated by the experiments of Lenay and his colleagues,
purely sensory substitution, without motor affordances to close
the perceptual interaction loop, may of limited use, since a userof the system may not have sufﬁcient information to associate
the sensation received to events in his or her surroundings. In tactile sensory substitution, this observation is supported by the poor
performance exhibited by users of tactile displays in the recognition of static forms, despite learning (Sampaio et al., 2001). By contrast, studies with closed-loop systems like the TVSS havedemonstrated that users are able to learn to improve their performance in object identiﬁcation tasks as they acquire sensorimotor
experience with the device (Bach-y-Rita and Kercel, 2003). However, the mechanism of this improvement (and of impairment
where static forms are concerned) is not altogether clear.
Motor control over sensory input allows to infer where theinformation is coming from, and to construct the space of the environment and events in it. Authors researching this subject haveclaimed that this information is encoded in sensorimotor contingencies that link sensation and control over the device (Lenayet al., 1997; O’Regan and Noë, 2001). Moreover, it enables the individual to sense more accurately, in the sense of spatial resolution,
than is possible with a static display. An extreme version of this‘‘hyperacuity” (Lenay et al., 2003) is exhibited by the system of Lenay and his colleagues, but it also applies to cases in which the sensor has a signiﬁcant resolution of its own.
The role of action in perception is a theme of current interest inperceptual psychology. It is related to active vision, exploratory
touch, and (to some degree) attentional listening. Eye movements
in vision provide a ready example. The sensitive foveal region of
the retina constantly pans across the visual scene in order to gather
knowledge about the regions of the environment that are mostsalient to whatever the user is doing. The work of Held and Hein
illustrated the importance of sensorimotor coordination to thedevelopment of an organisms intrinsic senses. Their famous experiments with early visually deprived kittens revealed that from apair of such kittens, one that was granted the same visual input
as the second, but deprived of the ability to exert motor control
over what it saw, failed to develop the visual capacity that the
self-moving kitten developed (O’Regan and Noë, 2001).
O’Regan and Noë have most prominently argued the case for asensorimotor basis of perception. They suggest that meaning insensation arises primarily from sensorimotor contingencies, suchas those cited above (O’Regan and Noë, 2001; Noë, 2005; Hurley
and Nöe, 2003). Noë writes: ‘‘Perception is not something that happens to us, or in us. It is something that we do.” The strong versionof this sensorimotor hypothesis can be said to hold more speciﬁcally that ‘‘sameness of perceptual experience” requires an identical sensorimotor proﬁle (Clark, 2006).
6. Evaluation, caveats, and open questions
The evaluation of sensory substitution devices is made more
difﬁcult, because the most interesting questions facing both scientists and device designers do not merely concern whether users ofthe devices are able to better perform tasks, but how a given levelof performance is achieved in relation to device design parameters
and the capacities of the user. Indications of performance are avail-able through behavioral or neurological data. For example, Ptito
and Kupers discuss differential activation of visual areas of thebrain in blind human subjects trained to perform an orientation
discrimination task via an electrotactile device applied to the tongue, where normally sighted subjects showed only activation in
the somatosensory cortex representing the tongue (Ptito et al.,2005). Psychoacoustic studies have evaluated users’ perceptual
abilities via visual depth perception tasks (Siegle and Warren,2007), object recognition, or using ophthalmological tests basedon character discrimination (Sampaio et al., 2001). Functional evaluations have sought to demonstrate that tasks such as assembly
using a teleoperated robot are performed more efﬁciently (Debus
et al., 2001), or that lipreading is improved (in both cases, with limited results).
6.1. Substitution – no substitute
Several authors have argued that, such evaluations aside, ‘‘sensory substitution” may not be appropriate as a description of theexperiences that such devices produce (Lenay et al., 2003). Theseexperiences fail to match those provided by intrinsic perceptual
channels in several respects. Visual to tactile substitution devices
might allow their users to recognize the forms, locations, or contours of familiar objects, but, to date, such interfaces do not convey
the color, level of resolution, or depth of gradations of contrast thatare conveyed by vision through the eyes. The devices fail, therefore,
to convey a large proportion of what one would normally associatewith the visual experience, and they do not capture the beauty of
the world. Their users are said to fail to respond emotionally to
evocative images, such as those of loved ones, when they are perceived through the devices. From this standpoint, sensory substitution devices might be better viewed as supplements to anindividual’s perceptual apparatus, rather than as replacements.
They allow their users to obtain more knowledge about the environment, which is not equivalent to that provided by intrinsic
and intact perceptual channels.
6.2. From tactile sensory substitution to enactive interfaces
If sensory substitution devices are to one day enable experiences comparable to what intrinsic perception affords, signiﬁcantadvances will be required. Electronic sensing and display components must become higher in ﬁdelity, and equally importantly,
they must be closely matched to the intrinsic sensory apparatus
that is linked to the display. In the case of tactile-visual sensory
substitution, which seems to have been the subject of more studies
than other kinds of tactile sensory substitution has, there seem tobe serious questions as to the level of veridicality that can beachieved, given the limited spatial and temporal information transmission rates that are possible via the surface of the skin. For cases
in which the peripheral sensory organ is damaged, sensory prosthetics, including cochlear and retinal implants, appear to more
promise. Nonetheless, an understanding of the fundamental mechanisms involved would be of potential beneﬁt to the engineering of
sensory substitution devices and sensory prosthetics.
Indications are, moreover, that the range of applications of sensory substitution systems in areas other than sensory aids will continue to grow. This article has provided numerous examples. They
already display the most important features of enactive interfaces,
including the timely delivery of sensory information, the central
role played by continuous action or movement, and the strong
interplay of perception and action. In addition, they illustrate speciﬁc features associated to tangibility and tactile feedback, and tosensing of environmental information, that are relevant for emerging classes of human computer interfaces. Finally, they raise distinct questions related to the possibility of designing for cross-

Please cite this article in press as: Visell, Y., Tactile sensory substitution: Models for enaction in HCI, Interact. Comput. (2008),
doi:10.1016/j.intcom.2008.08.004

ARTICLE IN PRESS
Y. Visell / Interacting with Computers xxx (2008) xxx–xxx

modal perception. A better understanding of the concepts and design considerations relevant for tactile sensory substitution systems, as summarized here, might therefore aid the creation andinvestigation of current enactive interfaces, and inﬂuence the design of future human computer interactions that have not yet been
conceived.
Acknowledgements
The author acknowledges support from the Natural Sciencesand Engineering Research Council of Canada and the Centre forInterdisciplinary Research in Music Media and Technology. The
author gratefully acknowledges the anonymous reviewers for theirhelpful comments, and Vincent Hayward, whose graduate seminaron haptics at McGill University provided the original impetus forthis article.
References
Adams, R.J., Hannaford, B., 1999. Stable haptic interaction with virtualenvironments. IEEE Transactions on Robotics and Automation 15 (3), 465–474.
Akinbiyi, T., Reiley, C.E., Saha, S., Burschka, D., Hasser, C.J., Kontarinis, D.A., Yuh, D.,
Okamura, A.M., 2006. Dynamic augmented reality for sensory substitution in
robot-assisted surgical systems. In: Proceedings of the 28th IEEE EMBS Annual
International Conference.
Amedi, A., Bermpohl, F., Stern, W., Camrpodon, J.A., Bermpohl, F., Merabet, L., Meijer,
P., Pascual-Leone, A., 2007. Extracting shape and location information conveyed
by visual-to-auditory sensory substitution activates the lateral occipitalcomplex and dorsal visual stream respectively in blind and sighted
individuals. In: IMRF 2007, the 8th Annual Meeting of the International
Multisensory Research Forum.
Auvray, M., 2004. Immersion et perception spatiale: L’exemple des dispositifs desubstitution sensorielle, Ph.D. thesis, Ecole des Hautes Etudes en Sciences
Sociales.
Auvray, M., Hanneton, S., Lenay, C., O’Regan, K., 2005. There is something out there:
distal attribution in sensory substitution, twenty years later. Journal of
Integrative Neuroscience 4 (4), 505–521.
Bach-y-Rita, P., 2001. Theoretical and practical considerations in the restoration offunction after stroke. Topics in Stroke Rehabilitation 8 (3), 1–15.
Bach-y-Rita, P., 2003. Seeing with the brain. International Journal of Human–
Computer Interaction 15 (2), 285–295.
Bach-y-Rita, P., 2004. Tactile sensory substitution studies, the coevolution of humanpotential and converging technologies. Annals of the New York Academy of
Sciences 1013, 83–91.
Bach-y-Rita, P., Collins, C.C., Sauders, F., White, B., Scadden, L., 1969. Vision
substitution by tactile image projection. Nature 221, 963–964.
Bach-y-Rita, P., Kaczmarek, K.A., Tyler, M.E., Garcia-Lara, M., 1998. Form perception
with a 49-point electrotactile stimulus array on the tongue: a technical note.
Journal of Rehabilitation Research and Development 35, 427–430.
Bach-y-Rita, P., Kercel, S.W., 2003. Sensory substitution and the human–machineinterface. Trends in Cognitive Sciences 7 (12), 541–546.
Bavelier, D., Nevile, H.J., 2002. Cross-modal plasticity: where and how? Nature
Reviews in Neuroscience 3, 443–452.
Benali-Khoudja, M., Hafez, M., Alexandre, J.-M., Kheddar, A., 2004. Tactile interfaces:
a state of the art survey. In: 35th International Symposium on Robotics, Paris.
Bliss, J.C., Katcher, M.H., Rogers, C.H., Shepard, R.P., 1970. Optical-to-tactile image
conversion for the blind. IEEE Transactions on Man–Machine Systems 11 (1),58–65.
Browse, R.A., McDonald, M.L., 1992. Using tactile information in telerobotics. IEEE
Transactions on Systems, Man, and Cybernetics 22 (5), 1205–1210.
Capp, M., Picton, P., 2000. The optophone: an electronic blind aid. Engineering
Science and Education Journal 9 (3), 137–143.
Cholewiak, R.W., Collins, A.A., 2000. The generation of vibrotactile patterns on alinear array: inﬂuences of body site, time, and presentation mode. Perception
and Psychophysics 62 (6), 1220–1235.
Chouvardas, V.G., Miliou, A.N., Hatalis, M.K., 2005. Tactile displays: a short overview
and recent developments. In: Proceedings of the 5th International Conference
on Technology and Automation (ICTA’05).
Clark, A., 2006. Cognitive complexity and the sensorimotor frontier. AristotelianSociety Supplementary 80 (1), 43–65.
Clifford, C.W., 2002. Perceptual adaptation: motion parallels orientation. Trends in
Cognitive Sciences 6 (3), 136–143.
Cohn, M.B., Lam, M., Fearing, R.S., 1992. Tactile feedback for teleoperation. In:
Telemanipulator Technology SPIE Proceedings 1833.
Collins, C.C., 1970. Tactile television: mechanical and electrical image projection.
IEEE Transactions on Man–Machine Systems 11, 65–71.
Craig, J.C., Johnson, K.O., 2000. The two-point threshold: not a measure of tactile
spatial resolution. Current Directions in Psychological Science 9 (1), 29–32.
Debus, T., Becker, T., DUpont, P., Jang, T.-J., Howe, R., 2001. Multichannel vibrotactile
display for sensory substitution during teleoperation. In: Proceedings of the

15

SPIE International Symposium on Intelligent Systems and Advanced
Manufacturing.
Debus, T., Jang, T.-J., Dupont, P., Howe, R., 2002. Multi-channel vibrotactile display
for teleoperated assembly. In: Proceedings IEEE International Conference on
Robotics and Automation.
Debus, T., Jang, T.-J., Dupont, P., Howe, R., 2004. Multi-channel vibrotactile display
for teleoperated assembly. Int. J. of Control, Automation, and Systems 2 (3),390–397.
Degenaar, M., Lokhorst, G.-J., 2005. Molyneuxś problem. In: Zalta, E.N. (Ed.), The
Stanford Encyclopedia of Philosophy. Stanford University, Fall.
Epstein, W., Highes, B., Schneider, S., Bach-y-Rita, P., 1986. Is there anything out
there? a study of distal attribution in response to vibrotactile stimulation.
Perception 15 (3), 275–284.
Foulke, E., 1982. Reading braille. In: Schiff, W., Foulke, E. (Eds.), Tactual Perception:
A Sourcebook. Cambridge University Press, pp. 168–208.
Fujimoto, I., Yamada, Y., Maeno, T., Morizono, T., Umetani, Y., 2004. Study on a‘‘tactile mirror” for displaying static friction sensation with tactile perception
feedback. In: Proceedings of the IEEE International Conference on Robotics andAutomation.
Gallace, A., Tan, H.Z., Spence, C., 2007. The body surface as a communication system:
the state of the art after 50 years. Presence 16 (6), 655–676.
Gibson, J.J., 1962. Observations on active touch. Psychological Reviews 69, 477–491.
Goldstone, R.L., 1998. Perceptual learning. Annual Review of Psychology 49, 585–
612.
Hafez, M., 2007. Tactile interfaces: technologies, applications, and challenges. VisualComputing 23, 267–272.
Hamilton, R.H., Pacual-Leone, A., 1998. Cortical plasticity associated with braillelearning. Trends in Cognitive Sciences 2 (5), 168–174.
Hart, S.G., Staveland, L.E., 1988. Development of nasa-tlx. In: Hancock, P.A.,
Meshkati, N. (Eds.), Human Mental Workload. North Holland Press, pp. 239–
250.
Hasser, C.J., 1993. Tactile Feedback For A Force Reﬂecting Haptic Display. Universityof Dayton. Master’s thesis.
Hayashi, K., Ninjouji, T., 2004. Two-point discrimination threshold as a function of
frequency and polarity at ﬁngertip by electrical stimulation. In: Proceedings ofthe 26th Annual International Conference of the IEEE EMBS.
Hayward, V., Astley, O.R., Cruz-Hernandez, M., Grant, D., Robles-De-La-Torre, G.,2004. Haptic interfaces and devices. Sensor Review 24 (1), 16–29.
Hein, A., Brell, M., 2007. Contact – a vibrotactile display for computer aided surgery.
In: Proceedings of the Second Joint EuroHaptics Conference and Symposium on
Haptic Interfaces for Virtual Environment and Teleoperator Systems (WHC’07).
Hurley, S., Nöe, A., 2003. Neural plasticity and consciousness. Biology & Philosophy
18, 131–168.
Iwamoto, T., Maeda, T., Shinoda, H., 2001. Focused ultrasound for tactile feeling
display. In: Proceedings of ICAT, Tokyo, Japan.
Jeka, J.J., Lackner, J., 1994. Fingertip contact inﬂuences human postural control.
Experimental Brain Research 100.
Jones, L.A., Sarter, N.B., 2008. Tactile displays: guidance for their design andapplication. Human Factors 50 (1), 90–111.
Kaczmarek, K., Webster, J.G., Bach-y-Rita, P., Tompkins, W.J., 1991. Electrotactile
and vibrotactile displays for sensory substitution systems. IEEE Transactions on
Biomedical Engineering 38 (1), 1–16.
Kadkade, P.P., Benda, B.J., Schmidt, P.B., Wall III, C., 2003. Vibrotactile display coding
for a balance prosthesis. IEEE Transactions on Neural Systems andRehabilitation Engineering 11 (4), 392–399.
Kajimoto, H., Kawakami, N., Tachi, S. 2003. In: Psychophysical evaluation of receptor
selectivity in electrotactile display, Proceeding of 13th International
Symposium on Measurement and Control in Robotics (ISMCR), vol. 13, pp. 8386.
Kajimoto, H., Inami, M., Kawakami, N., Tachi, S., 2004. Smarttouch: electric skin totouch the untouchable. IEEE Computer Graphics and Applications 24 (1), 36–43.
Kandel, E.R., Schwartz, J.H., Jessell, T.M., 2000. Principles of Neural Science, 4th ed.
McGraw-Hill.
Kitagawa, M., Dokko, D., Okamura, A.M., Yuh, D.D., 2005. Effect of sensory
substitution on suture-manipulation forces for robotic surgery systems.
Journal of Thoracic and Cardiovascular Surgery 129 (1), 151–158.
Kolb, B., Whishaw, I.Q., 1998. Brain plasticity and behavior. Annual Review of
Psychology 49 (1), 43–64.
Kontarinis, D.A., Howe, R.D., 1995. Tactile display of vibratory information inteleoperation and virtual environments. Presence 4 (4), 387–402.
Kreuger, L.E., 1970. David Katz’s der aufbau der tastwelt (the world of touch): a
synopsis. Perception and Psychophysics 7 (6), 337–341.
Kreuger, L.E., 1982. Tactual perception in historical perspective: David Katz’s worldof touch. In: Schiff, W., Foulke, E. (Eds.), Tactual Perception. Cambridge
University Press.
LaMotte, R.H., Whitehouse, J., 1986. Tactile detection of a dot on a smooth surface:
peripheral neural events. Journal of Neurophysiology 56 (5), 1109–1128.
Lenay, C., Canu, S., Villion, P., 1997. Technology and perception: the contribution of
sensory substitution systems. In: 2nd International Conference on Cognitive
Technology (CT’97), p. 44.
Lenay, C., Gapenne, O., Hanneton, S., 2003. Sensory substitution: limits andperspectives. In: Hatwell, Y., Streri, A., Gentaz, E. (Eds.), Touching for
Knowing: Cognitive Psychology of Haptic Manual Perception. John Benjamins.
Lenay, C., Gapenne, O., Stewart, J., 2001. Constitution of spatiality in relation to the
lived body: a study based on prosthetic perception. In: Symposium on

Please cite this article in press as: Visell, Y., Tactile sensory substitution: Models for enaction in HCI, Interact. Comput. (2008),
doi:10.1016/j.intcom.2008.08.004

ARTICLE IN PRESS
16

Y. Visell / Interacting with Computers xxx (2008) xxx–xxx

Emergence and Development of Embodied Cognition (EDEC2001), Beijing,
China.
Lévesque, V., 2005. Blindness, technology, and haptics, Tech. Rep. TR-CIM-05.08,
McGill University Centre for Intelligent Machines.
Lieberman, J., Breazeal, C., 2007. Tikl: Development of a wearable vibrotactile
feedback suit for improved human motor learning. IEEE Transactions on
Robotics 23 (5), 919–926.
Loomis, J.M., 1992. Distal attribution and presence. Presence 1 (1), 113–119.
Massimino, M.J., 1991. Using bisensory feedback displays for space teleoperation.
Acta Astronautica 24, 391–397.
Massimino, M.J., 1995. Improved force perception through sensory substitution.
Control Engineering Practice 3 (2), 215–222.
Merleau-Ponty, M., 1945. Phenomenology of Perception, 2002nd ed. Routledge,
New York.
Minsky, M., Ming, O.-Y., Steele, O., Brooks Jr., F.P., Behensky, M., 1990. Feeling and seeing:
issues in force display. ACM SIGGRAPH Computer Graphics 24 (2), 235–241.
Morgan, M.J., 1977. Molyneux’s Question. Cambridge University Press.
Noë, A., 2005. Action in Perception. MIT Press.
O’Regan, J.K., Noë, A., 2001. A sensorimotor account of vision and visual
consciousness. Behavioral and Brain Sciences 24, 939–1031.
Paré, M., Behets, C., Cornu, O., 2003. Paucity of presumptive rufﬁni corpuscles in theindex ﬁnger pad of humans. Journal of Comparative Neurology 456, 260–268.
Paré, M., Smith, A.M., Rice, F.L., 2002. Distribution and terminal arborizations of
cutaneous mechanoreceptors in the glabrous ﬁnger pads of the monkey. Journalof Comparative Neurology 445, 347–359.
Pascual-Leone, A., Amedi, A., Fregni, F., Merabet, L.B., 2005. The plastic human brain
cortex. Annual Review of Neuroscience 28, 377–401.
Pascual-Leone, A., Hamilton, R.H., 2001. The metamodal organization of the brain.
In: Casanova, C., Ptito M. (Eds.), Progress in Brain Research.
Pascual-Leone, A., Torres, F., 1993. Plasticity of the sensorimotor cortex
representation of the reading ﬁnger in braille readers. Brain 116 (1), 39–52.
Philipona, D., O’Regan, J.K., 2005. Perception of the structure of the physical worldusing unknown multimodal sensors and effectors. In: Proceedings of NIPS.
Philipona, D., O’Regan, K., Nadal, J.-P., 2003. Is there something out there? inferring
space from sensorimotor dependencies. Neural Computation 15 (9), 2029–2049.
Poiriera, C., Voldera, A.G.D., Scheiberb, C., 2007. What neuroimaging tells us about
sensory substitution. Neuroscience & Biobehavioral Reviews 31 (7), 1064–1070.
Ptito, M., Moesgaard, S.M., Gjedde, A., Kupers, R., 2005. Cross-modal plasticity
revealed by electrotactile stimulation of the tongue in the congenitally blind.
Brain 128 (3), 606–614.
Rath, M., Rocchesso, D., 2005. Continuous sonic feedback from a rolling ball. IEEE
MultiMedia 12 (2), 60–69.
Reed, C.M., Delhorne, L.A., 1995. Current results of a ﬁeld study of adult users oftactile aids. Seminars in Hearing 16, 305–315.
Roll, J.P., Gilhodes, J.C., 1995. Proprioceptive sensory codes mediating movement
trajectory perception: human hand vibration-induced drawing illusions.
Canadian Journal of Physiology and Pharmacology 73 (2), 295–304.
Sampaio, E., Maris, S., yRita, P.B., 2001. Brain plasticity: ‘visual’ acuity of blind
persons via the tongue. Brain Research 908, 204–207.
Siegle, J.H., Warren, W.H., 2007. Distance perception in visual-to-tactile sensory
substitution. In: Three Dimensional Sensory and Motor Space, Sant Feliu deGuixols, Spain.
Spirkovska, L., 2005. Summary of tactile user interfaces techniques and systems,
Tech. Rep. NASA/TM-2005-213451, National Aeronautics and Space
Administration, Ames Research Center, Moffett Field, California.

Starkiewicz, W., Kuliszewski, T., 1963. The 80-channel elektroftalm. In: Proceedingsof the lnternational Congress Technology Blindness, Am. Found. Blindness, New
York, NY, USA.
Tan, H.Z., Durlach, N.I., Reed, C.M., Rabinowitz, W.M., 1999. Informationtransmission with a multiﬁnger tactual display. Perception and Psychophysics
61 (6), 993–1008.
Tan, H.Z., Pentland, A., 1997. Tactual displays for wearable computing. PersonalTechnologies 1, 225–230.
Théoret, H., Merabet, L., Pascual-Leone, A., 2004. Behavioral and neuroplasticchanges in the blind: evidence for functionally relevant cross-modal
interactions. Journal of Physiology – Paris 98 (1-3), 221–233.
Torebjork, H.E., Schady, W., Ochoa, J., 1984. Sensory correlates of somatic afferent
ﬁber activation. Human Neurobiology 3 (1), 15–20.
Triggs, T.J., Levison, W.H., Sanneman, R., 1973. Some experience with ﬂightrelated electrocutaneous and vibrotactile displays. In: Geldard, F.A. (Ed.),
Conference Cutaneous Communication System Devices. Psychonomic Society,pp. 57–64.
van Erp, J.B.F., 2004. Vibrotactile in-vehicle navigation system, transportation
research part F. Human Factors 7, 247–256.
van Erp, J.B.F., 2005. Presenting directions with a vibrotactile torso display.
Ergonomics 48 (3), 302–313.
van Erp, J.B.F., 2005. Vibrotactile spatial acuity on the torso: effects of location andtiming parameters. In: Proceedings of the First Joint Eurohaptics Conference
and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator
Systems.
van Erp, J.B.F., 2006. Touch down: The effect of artiﬁcial touch cues on orientation in
microgravity. Neuroscience Letters 404, 78–82.
van Erp, J.B.F., van Veen, H.A.H.C., 2003. A multi-purpose tactile vest for astronauts
in the international space station. In: Proceedings of Eurohaptics.
van Erp, J.B.F., van Veen, H.A.H.C., Jansen, C., Dobbins, T., 2005. Waypoint navigation
with a vibrotactile waist belt. ACM Transactions on Applied Perception 2 (2),106–117.
van Erp, J.B.F., Verschoor, M.H., 2004. Cross-modal visual and vibrotactile tracking.
Applied Ergonomics 35, 105–112.
Vuillerme, N., Chenu, O., Pinsault, N., Moreau-Gaudry, A., Fleury, A., Demongeot, J.,
Payan, Y., 2007a. Pressure sensor-based tongue-placed electrotactile
biofeedback for balance improvement. In: Proceedings of the 29th Annual
International Conference of the IEEE Engineering in Medicine and Biology
Society.
Vuillerme, N., Chenu, O., Pinsault, N., Moreau-Gaudry, A., Fleury, A., Demongeot, J.,
Payan, Y. 2007b. Pressure sensor-based tongue-placed electrotactile
biofeedback for balance improvement - Biomedical application to prevent
pressure sores formation and falls. In: Proc. of the Engineering in Medicine andBiology Society.
Vuillerme, N., Chenua, O., Demongeota, J., Payana, Y., 2006. Improving human ankle
joint position sense using an artiﬁcial tongue-placed tactile biofeedback.
Neuroscience Letters 405 (1–2), 19–23.
Wall, S.A., Brewster, S., 2006. Sensory substitution using tactile pin arrays: humanfactors, technology, and applications. Signal Processing 86, 3674–3695.
Weiskrantz, L., 1998. Blindsight: A Case Study and Implications. Oxford University
Press.
White, B.W., 1970. Perceptual ﬁndings with the vision-substitution system. IEEE
Transactions on Man–Machine Systems 11, 54–59.
Yao, H.-Y., Hayward, V., 2006. An experiment on length perception with a virtual
rolling stone. Proceedings of Eurohaptics, 325–330.

Please cite this article in press as: Visell, Y., Tactile sensory substitution: Models for enaction in HCI, Interact. Comput. (2008),
doi:10.1016/j.intcom.2008.08.004