Noname manuscript No.
(will be inserted by the editor)

Spectrum-Based Synthesis of Vibrotactile Stimuli: Active
Footstep Display for Crinkle of Fragile Structures
Shogo Okamoto · Shun Ishikawa · Hikaru Nagano · Yoji Yamada

Received: date / Accepted: date

Abstract When a human crinkles or scrunches a fragile object, for which the yield force is very small that
it is hardly perceived, they identify the material of theobject based on tactile stimuli delivered to the skin. In
addition, humans are able to recognize materials evenwhen they are crinkled at different speeds. In order torealize these human recognition features of the crinkle
of a fragile object, we develop a vibrotactile synthesis
method. This method synthesizes the vibrotactile acceleration stimuli in response to a crinkle speed basedon the preliminarily measured acceleration spectra. Using this method, we develop an active footstep displaythat presents a virtual crinkle of fragile structures madeof different materials to its users. Experimental participants could identify three of the four types of virtual
structure materials at rates significantly higher than the
chance level. The four materials were copy and typing
paper, aluminum foil, and polypropylene film. Furthermore, the trends of answer ratios exhibit good correspondence with those for the real cylindrical fragile objects. We conclude that the developed method is valid
for the virtual crinkle of fragile structures and will enhance the validity of virtual reality systems, such as a
virtual walkthrough system.

Keywords amplitude spectrum · virtual material ·haptic interface

This work was partly supported by MEXT KAKENHI
23135514 and the Hori Sciences and Arts Foundation.
The authors are with the Department of Mechanical Scienceand Engineering, Graduate School of Engineering, Nagoya
University, Japan.
E-mail: see http://www.mech.nagoya-u.ac.jp/asi/en/

1 Introduction
Force and tactile feedback associated with the fractureof structures or materials is necessary for the haptic
simulation of the manipulation of fragile materials or
surgical and sculptural procedures. For example, methods for displaying reaction forces when cutting soft tissue with scissors (Fujino et al. 2008; Okamura et al.
2003; Wakamatsu et al. 1998) or when burring or cutting teeth or bones have been studied (Wang et al.
2005; Agus et al. 2003; Arbabtafti et al. 2011). In thecase of fragile structures, for which the yield forces areso small that they are hardly perceived, tactile stimuli
have a greater perceptual importance than force stimuli. Based on these tactile stimuli, humans can detect
the fracture and identify the types of broken structures
or materials. To date, however, tactile feedback methods for fractures have rarely been studied, as most of
the related research has been focused on displaying theassociated forces.
The objective of this study is to develop a vibrotactile display method for the crinkle or scrunch of fragile structures. The tactile stimuli associated with thecrinkle depend on the type of material or the speed of
the crinkling motion. Hence, this study aims to produce
tactile stimuli in accordance with the active movements,
especially the velocities, of the humans who are crinkling the structures. Moreover, the goal of this method
is the presentation of stimuli that would enable humans
to identify the material types of the structures.
Although it is typical to present tactile stimuli tohuman hands and fingers, we present stimuli to thesoles of the feet. A synthesizing method for vibrotactile stimuli acquired when breaking or scrunching fragile objects on a floor is developed. These stimuli are
expected to enhance the enjoyment and effectiveness2of virtual walkthrough systems (Chim et al. 2003; Rafferty et al. 1998), including such things as fallen leaves
or frost on the ground. Walkthrough systems have been
partly studied for the rehabilitation of those with walking disabilities. For such purposes, perceptual feedback
to the sole is considered to be important. The application of the technique to hands is also beneficial for
haptic simulation involving fragile objects such as paper craft or shell-like objects. Examples of such simulations typically include the training simulator for traditional craft. In these hand manipulation tasks, the senseof crinkling structures is imparted to the users mainly
when they fail in tasks because of the application ofexcessive forces to the structures. On the other hand,stimuli to the foot soles are presented to the users inevery step to ensure quality simulation. From this aspect, both hands and soles are comparably important
application points.
The presentation of virtual reality stimuli to thesoles has seldom been attempted. For example, Visellet al. (2008) applied a sound synthesis model for walking to vibrotactile stimuli to the soles in order to present
a sense of virtual floor materials with corresponding
sounds and graphics (Law et al. 2008). In addition, they
investigated identifiable vibrotactile stimuli patterns forfeet while standing still (Visell et al. 2009). However,this research did not focus on the tactile stimuli associated with the fracture of structures. For purposes otherthan virtual reality, many researchers are interested in
the presentation of stimuli to the soles. Velázquez et al.
(2009) arranged vibrators in the soles of shoes and studied the human perception of spatial stimuli patterns;
they also made use of their device for navigation while
walking. Watanabe et al. (2005) attempted to navigate
walking cycles using shoes with vibrotactile feedback.
Such feedback for the feet has been provided for gait
training (Lurie et al. 2011). It has been reported thatvibratory stimuli to the soles can affect the center ofgravity, or its awareness, of a person standing upright(Kavounoudias et al. 1999; Maki et al. 1999; Roll et al.
2002).
This article is based on the study of Okamoto etal. (2011), where the synthesis of vibrotactile stimuli
for the crinkling of fragile structures was originally introduced. In the previous work, human capability toidentify the synthesized stimuli among three types ofmaterials was investigated. For further generality, we
conduct experiments involving four types of materials
by recruiting a large number of new participants. The
major difference between the previous and present articles lies in the fact that we experimentally compare
the correct and confusing answer ratios in identification
tasks between real and virtual stimuli of fragile objects.

Shogo Okamoto et al.

The comparison helps validate the synthesizing methodon the basis of the similarity between the synthesizedstimuli and the real stimuli.
2 Observation of Acceleration Signals during
Crinkle of Fragile Structures
To develop a vibrotactile stimuli model, we measured
the acceleration stimuli generated by the crinkle of fragile structures. We adopted cylinders made of copy (Kankyo Yoshi A4; RISO Kagaku Corp., Tokyo, Japan; weight:
65 g/m2 ; thickness: 80 µm) and typing (Typing Pad
10N; Kokuyo Co. Ltd., Osaka, Japan; weight: 50 g/m2 ;thickness: 35 µm) paper, aluminum foil (aluminum foil
0203; Toyo Aluminum Ekco Products Co. Ltd., Osaka,
Japan; weight: 30 g/m2 ; thickness: 11 µm), and polypropylene film (ST 13-30; Frontier Co. Ltd., Osaka,
Japan; weight: 27 g/m2 ; thickness: 30 µm) (Fig. 1a)as the fragile structures. These structures crinkle without producing noticeable reaction forces. Each cylinder
had a diameter of 40 mm and a height of 20 mm. Fig. 2
shows a typical scrunch of one of the paper cylinders.
It is crinkled, or scrunched, rather than fractured. As
shown in Fig. 1b, a human wearing socks but no shoes
stepped on and scrunched the structure on a 2 mm
thick acrylic panel. The panel was part of the footstep display, which is discussed later. An accelerometer(ADXL335; Analog Devices, MA; bandwidth: 550 Hz)
attached to the other side of the panel just beneath thestructure, measured the acceleration when the structure
was scrunched. The bandwidth of the accelerometercovered the frequency of our interest, which was up to
500 Hz. The peak human sensitivity for vibrotactile acceleration stimuli lies at approximately 100 Hz for acceleration stimuli (Morioka et al. 2005). Frequencies over500 Hz are hardly effective in this study, because theseaccelerations are smaller than the perceptual thresholds for all the materials, which are at least 0.3 m/s2
(Morioka et al. 2005). A potentiometer (LP-200FJ; Midori Precisions Co. Ltd., Tokyo, Japan) fastened to theparticipant’s ankle measured the position of the footalong the Z-axis. The ankle joint was bound to a metal
plate by two fabric bands such that the sole remained
parallel to the panel (Fig. 1c). The foot position and acceleration caused by the crinkle were recorded at 5 kHz.
Fig. 3 shows the acceleration observed along the
Z-axis when the paper cylinder was crinkled. Figs. 3aand 3b show those when the foot velocity was 50 and
200 mm/s, respectively. These velocity values are averages for the time that the foot was in contact withthe cylinder and moving downward over 10 mm. The
foot came into contact with the cylinder at t = 0 in the
figures. Even when the same cylinder was broken, theSpectrum-Based Synthesis of Vibrotactile Stimuli: Active Footstep Display for Crinkle of Fragile Structures

3

Fig. 3 Acceleration along Z-axis caused by fracture of copypaper cylinder. a) Acceleration with foot velocity of 50 mm/s.
b) Acceleration with foot velocity of 200 mm/s.
Fig. 1 Observation of acceleration stimuli due to the crinkleof fragile objects. a) Cylindrical fragile object. b) Side viewof setup. c) Bird’s eye view of setup.

Fig. 4 Acceleration spectra for fracture of copy-paper cylinder.
Fig. 2 Crinkled copy-paper cylinder (typical case).

3 Spectrum-based Vibrotactile Synthesis
Method for Crinkle of Fragile Structures
3.1 Synthesis Method
acceleration profiles, including the magnitudes or decay times, varied with the contact velocity. Fig. 4 shows
the amplitude spectra of the accelerations in Figs. 3aand 3b. These spectra differ depending on the velocity. For example, when the foot velocity was 50 mm/s
(Fig. 4a, bold), spectrum peaks were found at around20, 70, and 280 Hz. In contrast, when the velocity was
200 mm/s (Fig. 4b, fine), the peaks were around 10,
40, and 180 Hz. The magnitude across the gross frequency bands was larger for the velocity of 200 mm/s
than for the velocity of 50 mm/s. These data suggestthat velocity-dependent tactile stimuli are required todisplay the crinkle of fragile structures. It should be
noted that the above-mentioned accelerations included
vibrations in the experimental setup, such as the top
panel, in addition to those in the cylinder.

The amplitude, or power spectrum, determines the subjective quality of vibrotactile stimuli, and phase information for the components does not significantly affect perception (Bensmaı̈a et al. 2000; Cholewiak et al.
2010; Wiertlewski et al. 2011). Hence, we also focus on
the amplitude spectrum of the vibrotactile stimuli. We
acquire the spectrum of the acceleration by computing
the discrete cosine transformation (DCT) which provides the amplitude spectrum.
Let yi (i = 0, . . . , N − 1) denote the temporal sequence of acceleration caused by the crinkle of a structure. The spectrum of yi is determined by the expression
N −1

Ck =

πk(i + 12 )2 X
) (k = 0, . . . , N − 1), (1)yi cos (
N i=0
N

4

Shogo Okamoto et al.

where Ck is the acceleration at the frequency (F k)/N .
F and N are the sampling frequency of the accelerationand the number of samples, respectively: F = 5000 Hz
and N = 2500. This is a type-II DCT with a factorof 2/N , which makes the DCT indices the amplitudesof the frequency elements. We measure the acceleration
three times for a single condition and average the spectra to obtain the acceleration spectrum of the crinkle.
As described above, the acceleration sensed by theaccelerometer includes vibrations in the experimental
setup, mainly those of the top panel, in addition to
those from the crinkle of the structure. Hence, in order
to separate these accelerations, spectra subtraction is
performed. Ck,panel,obj is the spectrum of the accelerations that are observed when a structure is scrunched.
Ck,panel is the spectrum of the accelerations observed
when the top panel is stepped on without a structure
being present. Thus, we determine the spectrum of theaccelerations that come from the crinkle of the structure( as
Ck′ = |Ck,panel,obj | − |Ck,panel |Ck′ = 0(2)

if |Ck,panel,obj | − |Ck,panel | < 0.

The acceleration spectrum depends on the foot velocity. In order to present the difference in the spectrum caused by changes in foot velocity, the spectrum
is synthesized by the interpolation or extrapolation of
the pre-measured spectra. The amplitude spectrum forvelocity v is determined by|Ck′ (vi+1 ) − Ck′ (vi )|Ck′ (v) =(v − vi ) + Ck′ (vi )vi+1 − vi·sign(Ck,panel,obj (vj ))(3)(vi < v < vi+1 ), (k = 0, . . . , N − 1),vi if v − vi ≤ vi+1 − v(4)vi+1 otherwise.
These equations are used to determine the spectrum forv from the interpolation of the spectra for vi and vi+1 .
Initially, the spectra for vi were measured. These vi values were (v1 , v2 , v3 , v4 ) = (50, 100, 200, 300) mm/s. For
most cases of the natural steps in the experiments described later, the participants’ foot speed was within
this range. In the case of v > v4 or v < v1 , Ck′ (v) was
extrapolated. The signs of the frequency components
were the same as those of vi or vi+1 that was close
to v. This synthesized spectrum lost the phase information. However, as described above, the phase information does not significantly affect the perception ofvibrotactile stimuli.
Ck′ (v) is re-transformed into the temporal sequenceof accelerations. These are determined byN−1X
πk(i + 21 )) (i = 0, . . . , N − 1). (5)Ck′ (v) cos (
yi′ =
N
vj =

k=1

These yi′ values are applied to human soles as the acceleration stimuli for foot velocity v.

3.2 Example of Synthesized Acceleration Stimuli
Examples of the above process are shown in Fig. 5.
Fig. 5a shows the acceleration observed when the real
copy-paper cylinder was stepped on at 100 mm/s. Fig. 5bshows the acceleration spectra of Fig. 5a, as well asthat observed when the top panel was stepped on at
100 mm/s in the absence of the cylinder. The subtraction of these two spectra, following (2), gives the acceleration spectrum that comes from the virtual crinkleof the cylinder. Fig. 5c shows the acceleration acquiredby computing the inverse DCT of this subtracted spectrum. Fig. 5d shows the spectra of the virtual crinkles of
the copy paper at 100 and 200 mm/s. The acceleration
stimuli of a virtual crinkle at speeds between 100 and
200 mm/s is synthesized from these two spectra using(3). The synthesized acceleration for 150 mm/s is shownin Fig. 5e. For comparison, Fig. 5f shows the acceleration of a virtual crinkle at 200 mm/s. The time-domain
profile of the acceleration for 150 mm/s is closer tothat for 100 mm/s (Fig. 5c) than to that for 200 mm/s
(Fig. 5f). This is because the phase information or thesign of each frequency component, of the spectrum for
150 mm/s is the same as that for 100 mm/s, according
to (4). The vibrotactile display discussed later is used topresent acceleration stimuli of the virtual crinkle that
do not include the vibrations of the top panel. Acceleration in the range t < 0 in Fig. 5c, 5e, and 5f is notpresented to the users; only that for t ≥ 0 is available
to them.

3.3 Validation of the Method
The assumption of the linearity of spectra is likely to
be valid only for a limited range on contact speeds. In
the present study, the values of 50 and 100 mm/s are
small enough for the assumption to hold. Fig. 6 shows
an example of virtual synthesized spectrum of copy paper (gray) for 150 mm/s. That was synthesized using
those for contact speeds of 100 and 200 mm/s. For each
speed level, the spectra were the averages of those of
three samples. Another spectrum labeled as real crinkle (black) was computed on the basis of real accelerations recorded for 150 mm/s. That was also the averageof spectra of three samples. The spectrum originatingfrom the top panel was subtracted from the spectraof these real and virtual crinkles. Both real and synthesized virtual spectra look similar to each other. In

Spectrum-Based Synthesis of Vibrotactile Stimuli: Active Footstep Display for Crinkle of Fragile Structures

5

Fig. 6 Comparison between real and virtual crinkles at thecontact speed of 150 mm/s. Both spectra do not include thosefrom the top panel.

Fig. 7 Frequencies showing which synthesized spectra were
judged to be most similar to the observed ones. Left up: Copy
paper. Right up: Aluminum. Left down: Typing paper. Right
down: Polypropylene.

order to check whether synthesized spectra are similar to real ones, we systematically compared the observed (real cylinders) and synthesized (virtual cylinders) spectra. The cylinders were stepped on at contact
speeds of 150, 250, and 400 mm/s; they were steppedon ten times for each speed level. We then compared
the dissimilarity between the two spectra Si and Sj byusing the expression
dij =

500
X

|Si (fk ) − Sj (fk )|,

(6)

fk =0

Fig. 5 Example of synthesized acceleration data. a) Acceleration acquired by crinkle of copy paper on top panel at
contact speed of 100 mm/s. b) Spectra of Fig. 5a and acceleration acquired by stepping on top panel without cylinder at
100 mm/s. c) Acceleration stimuli of virtual crinkle of copy
paper when crinkled at 100 mm/s. d) Spectra of accelerationsof virtual crinkles at 100 and 200 mm/s. e) Synthesized acceleration stimuli of virtual crinkle at 150 mm/s. f) Acceleration
stimuli of virtual crinkle at 200 mm/s.

where Si (fk ) is the magnitude of acceleration for frequency fk in spectrum Si . Fig. 7 shows the numberof trials for which the observed spectra were judged
to be most similar to the synthesized ones. For example, for six of the ten trials of copy-paper cylinders (left-up bar in each cell), the spectra observed for
150 mm/s were judged to be most similar to those synthesized for 150 mm/s (single asterisk). For three trials,
the observed spectra were categorized as corresponding
to those synthesized for 100 mm/s (double asterisks).
Hence, the spectra observed for 150 mm/s were similar
to those synthesized for the same or close speed levels.
The same trends were observed for 250 and 400 mm/s
for all types of materials. These results indicate that

6

Shogo Okamoto et al.

the synthesizing algorithm produced spectra similar to
those observed for the same contact speeds.

4 Vibrotactile Footstep Display: Experimental
Equipment
We developed a vibrotactile footstep display in orderto present the user with a tactile sense of the crinkleof fragile structures. It applied vibrotactile stimuli tohuman soles. The display presented non-spatially distributed acceleration stimuli to the entire sole. We usually acquire spatially distributed stimuli on our soles
when stepping on objects. In this study, however, weused the simplified vibrotactile display in order to test
the validity of the spectrum-based vibrotactile synthesis method. This condition is similar to that existing
when we step on objects while wearing shoes, where we
receive the vibrotactile stimuli on the entire sole rather
than in a small area. Because we assumed that the fragile structures did not produce noticeable reaction forces
during crinkling, the device did not have a force display
function.
Hardware: The device used voice coil speakers (23 W,
4 Ω) as vibrotactile stimulators (Fig. 8). Four voice coils
were used, and an acrylic top panel was placed on them.
Springs were positioned beneath the voice coils to pushthem tightly against the top panel. The top panel was
fastened to four metal poles with screws, and was deformed by a few millimeters when stepped on by participants. Fabric tape was used to fix each user’s ankle
joint to a linear slider with a potentiometer (Fig. 1c).
The potentiometer measured the position and speed of
the foot in real time. The speed measured was the average speed of the foot while it moved from 20 to 10 mmabove the top panel. A computer using real-time Linux
received the output from the potentiometer and synthesized the stimuli, and it then sent a voltage input tothe voice coils at a refresh rate of 5 kHz. An amplifier(MAX9704; MAXIM Integrated Products, CA; 10 W)
was inserted between the computer and the voice coils.
Usage: A user wearing socks but no shoes sat in a chairand stepped on the top panel. The socks were used forprotecting the soles in case the acrylic top panel broke.
They stepped such that the arch of the foot came downon the center of the panel. According to the foot speed
measured by the potentiometer, the vibrotactile acceleration stimuli determined by the method in section 3
were produced. The footstep display started the stimuli
just as the participant’s foot reached the top panel.

Fig. 8 Foot step display based on voice-coil actuators.

Frequency shaping: In order to realize the acceleration
stimuli on the sole, the voltage inputs to the voice coils
were determined based on the frequency response curveof the experimental equipment. We measured the frequency response of the equipment while a load of 40 N
was applied to the top panel. This load was close to
the foot force of a typical user. Fig. 9 shows the frequency response of the equipment when the voltage inputs to the voice coils were 1.0 Vrms. The peak-to-peakacceleration was measured by an accelerometer placedat the center of the top panel. It can be seen to be
nonlinear with a peak frequency of around 360 Hz. We
also measured the response for voltage inputs of 0.1–
1.5 Vrms. The acceleration outputs of the equipmentchanged almost linearly with the voltage inputs when
the frequency was fixed. We prepared a weighting function W (f, a) that determined the applied voltage to thevoice coils from frequency f and desired acceleration
output a based on the frequency response of the experimental equipment. This function was almost a look-up
table; however, it determined the output voltage by interpolating between the nearest values of f and a in
the table. The acceleration spectrum was weighted by
W (f, a), and the inverse DCT was then used to transform the spectrum into a temporal sequence of voltage
data that was applied to the voice coils. The voltage
data were given by
ei =

N
−1
X
k=1

W(

πk(i + 21 )Fk ′, Ck (v)) cos (
)
N
N

(7)

(i = 0, . . . , N − 1).

This voltage input was used in the experiments described in section 5.

5 Experiment: Identification of Virtual andReal Material
In order to evaluate to what extent humans can identify
types of virtual material, we conducted identification
tasks. Participants undertook two types of experiments
in a balanced order. In both tasks, they were asked

Spectrum-Based Synthesis of Vibrotactile Stimuli: Active Footstep Display for Crinkle of Fragile Structures

Fig. 9 Frequency response of experimental system with voltage input of 1.0 Vrms (frequency vs. peak-to-peak acceleration output).

7

Fig. 11 Correct answer ratios for each material in Tasks 1
and 2.

ticipants listened to pink noise through noise-cancelingheadphones. The volume of the noise was adjusted by
the participants such that they did not hear the sounds
produced by the voice coils.

Fig. 10 Experimental setup. Participant sitting on chair andstepping on vibrotactile foot-step display. Left: Foot is held
up. Right: Foot placed on top panel.

to respond by stating which of the four materials—
copy and typing paper, aluminum foil, or polypropylene
film—was being crinkled underfoot. The number of material types was determined on the basis of the timerequired for the tasks. With these four materials, participants took approximately 60–80 min for performing
all procedures. Task 1 subjected them to the virtual
stimuli that mimicked each of these materials, while in
Task 2 they responded to the real cylindrical structures.
We then analyzed the correct and confusing answer ratios from each task and compared the results of the twotasks. The procedures and results are elaborated in thefollowing sections.

Common procedures: Two tasks were performed by theparticipants. Half of them performed Task 1 first, andthen, after a short break, they performed Task 2, andvice versa. All 15 participants performed Task 1, but
only ten of them performed Task 2. Before each task,
they were allowed to crinkle the four types of real cylindrical structures on the top panel. They experienced
this until they felt familiar with the stimuli, which tookapproximately 3–5 min.
Task 1 (virtual stimuli): In Task 1, the participants
performed an identification task of the virtual stimuli.
They were not provided with any information about
the virtual stimuli or their true correspondence to thereal materials. They stepped on the virtual stimuli thatwere randomly presented to them. After each trial, theyorally reported whether the type of material that theyfelt was copy or typing paper, aluminum foil, or polypropylene film. We did not specify the foot speeds.
Thus, they stepped on the top panel with arbitrary footspeeds, as they wished. Each type of virtual material
was presented ten times, giving a total of 40 stimuli ortrials per participant.

5.1 Tasks and ParticipantsParticipants: Fifteen university students were recruitedand paid by methods approved by the ethical committee of the School of Engineering in Nagoya University.
None of the participants had any experience with thestudy of vibrotactile displays. In order to avoid the possibility of falling and fatigue, they sat in a chair to participate in the experiments, as shown in Fig. 10. In order to shut out the sounds produced by the crinkle ofreal cylinders or by the voice coil speakers, the par-Task 2 (real stimuli): In Task 2, the participants identified the material of a real cylinder that was locatedbeneath the foot arch. A thin panel was placed on
the cylinder, and the participants crinkled the cylinder through the panel. Without this panel, the participants would be able to feel the edge of the cylinder on
the foot arch. In order to prevent this sensation frombeing perceived and to confine the perceivable stimuli
to the non-spatially distributed tactile stimuli, we used
this panel. The introduction of the panel approximated8

Shogo Okamoto et al.

Table 1 Answer ratios of Task 1 (virtual material): average ± standard deviation.

Presented
material

Copy paper
Aluminum foil
Polypropylene
Typing paper

Copy paper
0.79 ± 0.24∗∗∗0.033 ± 0.048
0.013 ± 0.051
0.007 ± 0.026

Participants’ answer
Aluminum foil
Polypropylene
0.047 ± 0.074
0.053 ± 0.14
0.39 ± 0.20∗
0.21 ± 0.16
0.19 ± 0.24
0.65 ± 0.27∗∗∗
0.48 ± 0.26
0.23 ± 0.22

Typing paper
0.11 ± 0.18
0.36 ± 0.23
0.12 ± 0.11
0.29 ± 0.15*, **, and *** indicate that the center of the distribution is significantly higher thanthe chance level of 0.25 at the significance level of 0.05, 0.01, and 0.001, respectively.
Table 2 Answer ratios of Task 2 (real stimulus): average ± standard deviation.

Presented
material

Copy paper
Aluminum foil
Polypropylene
Typing paper

Copy paper
0.97 ± 0.07∗∗∗
0.02 ± 0.04
00.01 ± 0.03

the tactile stimuli acquired by the real cylinder to thoseby the virtual stimuli, which made the answer ratios of
the two tasks comparable under similar conditions. Thispanel was used only in Task 2. Each material was again
presented ten times, giving 40 trials per participant.
5.2 Results
Task 1 (Virtual Stimuli): Table 1 and Fig. 11 show
the response and correct answer ratios in Task 1, where
the participants responded to the virtual materials. The
standard deviations were calculated among the participants. Copy paper exhibited the highest correct answer ratio (gray cells) of 0.79. This value was higher
than the chance level of the answer ratios, which was
0.25 (t0 (14) = 8.71, P = 5.02 × 10−7 , two-tailed). The
average correct answer ratios for aluminum and polypropylene were 0.39 (t0 (14) = 2.75, P = 0.016, twotailed), and 0.65 (t0 (14) = 5.79, P = 4.64 × 10−5 , twotailed), respectively. These values were also higher thanthe chance level. However, the correct answer ratio fortyping paper was 0.29, and so there was no significant
difference observed between this and the chance level(t0 (14) = 0.97, P = 0.35, two-tailed). The most confused pair was aluminum and typing paper. The participants did not differentiate between these two types ofmaterials at all. An ANOVA did not indicate significant
differences between the answer ratios in which the participants correctly responded to aluminum (0.39±0.20)
and to typing paper (0.29 ± 0.15), and those in which
they wrongly responded to aluminum (0.36 ± 0.23) andto typing paper (0.48±0.26) (ANOVA, F0 (3, 56) = 2.12,
P = 0.11).
Task 2 (real stimuli): Table 2 and Fig. 11 show theaverage correct and confusing answer ratios in Task 2,

Participants’ answer
Aluminum foil
Polypropylene
0.01 ± 0.03
0.01 ± 0.03
0.51 ± 0.22∗∗∗0.22 ± 0.20
0.15 ± 0.08
0.63 ± 0.25∗∗∗
0.36 ± 0.18
0.15 ± 0.17

Typing paper
0.01 ± 0.03
0.25 ± 0.18
0.22 ± 0.16
0.48 ± 0.20∗∗∗

where the participants responded to the real cylinders.
For all types of materials, the correct answer ratios
were higher than the chance level (copy paper: t0 (9) =10.67, P = 4.96 × 10−16 ; polypropylene: t0 (9) = 1.52,
P = 3.90 × 10−5 ; aluminum foil: t0 (9) = 1.19, P =4.03×10−4 ; typing paper: t0 (9) = 1.16, P = 5.20×10−4 ;all were two-tailed). The average correct answer ratio
for the copy paper was 0.97, again scoring the highest among the four, followed by polypropylene (0.63),
aluminum foil (0.51), and typing paper (0.48). Similar
to Task 1, typing paper and aluminum were often confused. With respect to all the correct answer ratios for
Tasks 1 and 2, those in Task 2 (real material) were significantly higher than those of Task 1 (virtual material)
according to a two-way ANOVA with the two factors
being the material and task type (F0 (1, 92) = 7.49,
P = 7.43 × 10−3 ). We computed a reliability index
based on the results of this two-way ANOVA. The intraclass correlation coefficient known as a reliability index
that indicates the valiance among participants was 0.97,
which means that individual differences were relatively
small compared with differences among the task types
and materials.
Summary: Regarding the overall correct answer ratios in the material identification tasks, those for thevirtual materials were smaller than the ratios for thereal materials. However, the participants could identify
three of the four material types at ratios significantly
higher than the chance level. The method presented
the characteristics of the real materials to some extent.
Furthermore, the method reproduced the mutual similarity of materials. The trends in the correct and confusing answer ratios were similar for both the real andvirtual stimuli. The difficulty of discrimination between
aluminum and typing paper was observed for both theSpectrum-Based Synthesis of Vibrotactile Stimuli: Active Footstep Display for Crinkle of Fragile Structures9

real and virtual materials. In addition, the correct answer ratios were in the same order for both the realand virtual materials. Although it is difficult to evaluate how realistic the virtual materials were, the answer
ratios of the identification tasks are good indicators ofthe validity of the developed method. In terms of thecorrect answer ratios, and the similarity in the answer
trends of identification tasks between real and virtualmaterials, the developed method is fairly good as a firstdisplay system for the crinkling of fragile objects.
6 Discussion
6.1 Similarity in Material Spectra
We now attempt to interpret the confusing answer ratios on the basis of the similarity between the spectraof materials. Fig. 12 shows the acceleration spectra acquired by scrunching the four types of real materials
at 200 mm/s. These spectra are the average of three
trials. Although a few indices for subjective similarity based on the spectra of high-frequency vibrotactile stimuli has been proposed by Bensmaı̈a and Hollins(2005), Bensmaı̈a et al. (2005), Israr and Tan (2006),
Okamoto and Yamada (2011), these indices are limited
to the vibrotactile textures or handle vibrations. The
validity of the index for transient stimuli such as theone discussed in the present article has been unknown.
Therefore, we purely compare the similarity among thespectra on the basis of the sum of amplitude differencesin the frequency bands by using (6).
Table 3 shows the dissimilarity scores between eachpair of materials. The values in the table were normalized by the highest dissimilarity score. The highest score(1.0) came from the copy paper/polypropylene pairing.
This was followed by copy paper/aluminum (0.74) andcopy paper/typing paper (0.79). Fig. 13 shows the correspondence between the dissimilarity scores and theconfusing answer ratios shown in Table 2 (Task 2). An
increase in the dissimilarity scores indicates that thetwo materials became less confused, which means thatthe results of the identification tasks are consistent withthese scores.
For example, the scores indicate that copy paper
is comparatively dissimilar to other materials. According to Table 2, copy paper was rarely confused withthe other materials. From Table 3, we see that typing paper and polypropylene are moderately similar,
with a dissimilarity score of 0.32. As can be seen in
Table 2, the participants confused these materials 15%
and 22% of the time. Finally, the dissimilarity scoressuggest that aluminum is similar to typing paper, witha score of 0.25. These materials were most frequently

Fig. 12 Amplitude spectra of the acceleration acquired forthe crinkle of real cylinders at 200 mm/s.
Table 3 Normalized dissimilarity score between two spectra.
The higher scores indicate that two materials are less similar.
Copy pap.
Aluminum
PP.

Aluminum
0.74
-

PP.
1.00
0.29
-

Typing pap.
0.79
0.25
0.32

Fig. 13 Confusing answer ratios and dissimilarity scores.
Correspondence between Tables 2 and 3.

confused in the experiments. The confusion ratios between aluminum and typing paper were 36% and 25%(Table 2).
From the above comparisons, there is good correspondence between the answer ratios and the spectrumbased dissimilarity scores for the materials. This correspondence supports the validity of the spectrum-based
design and discussion of vibrotactile synthesis method.

6.2 Future Work
According to the introspective reports collected after
the tasks, we speculate that the major cause of deterioration in the correct answer ratios for the virtual
stimuli was a temporal gap in the vibrotactile stimuli.
When the participants stepped on the real cylinders
placed on the top panel, the scrunching stimuli weredelivered to their foot soles just before the direct con-

10

tact between the sole and the top panel. However, inthe case of the virtual materials, the vibrotactile stimuli
and those caused by contact with the top panel were delivered simultaneously. Participants reported that thisconcurrent experience of two types of stimuli impeded
their identification performance. One approach to thisproblem is to install the stimulator on the user rather
than on the apparatus. For example, the introductionof a stimulator into the shoe sole would enable them toexperience the virtual stimuli before the foot contacts
the ground.
Though the aim of the synthesizing method developed in this study was to retain the similarity of acceleration spectra of the vibrotactile stimuli, we should also
pay attention to the similarity of stimuli in the time domain. In the case of amplitude-modulated vibrotactile
stimuli, the similarity in the time domain can be more
important for perceptual similarity of the stimuli thanthat in the power spectra (Park et al. 2011). In order toretain the profile of the stimuli in the time domain, thesynthesis of the phase spectrum should be introducedin the method. This is also an interesting topic for a
future study.
The algorithm used in the present study was indifferent to the weight of the foot; it assumed that theeffects of the weight were insignificant. Fragile structures crinkle without large reaction forces. Hence, themomentum of the footstep was not exchanged with thestructure. A large part of the momentum was received
by the top panel on which the structure was positioned.
The effects of human weight on the virtual stimuli were
supposedly small because the algorithm separated theaccelerations of the structure and floor. However, thelimitation of this assumption should be examined in
the future.
7 Conclusion
This study developed a vibrotactile display method to
capture the crinkle of fragile structures. For such fragile
structures, people identify the materials depending on
the tactile stimuli delivered to their skin rather thanthe hardly perceivable reaction forces. Tactile stimuli,
which depended on the velocity of the crinkle movement, were realized by synthesizing the acceleration
spectra recorded beforehand. We presented stimuli associated with the crinkle of fragile cylindrical objects
to human foot soles by using this synthesizing method.
The objects were made of four materials—copy and typing paper, aluminum foil, and polypropylene film. The
vibrotactile display for the soles was built using voicecoil speakers. We validated the synthesizing methodby conducting a material-identification experiment with

Shogo Okamoto et al.

the synthesized virtual stimuli. The experimental participants could identify three types of materials, theexception being typing paper, at rates higher than the
chance level. We also compared the participants’ responses to the synthesized stimuli with those to thecrinkle of real cylindrical structures. Although the overall correct answer ratio for the synthesized stimuli wassmaller than that for the real stimuli, similar trends
were observed among the correct and confusing answer ratios. For both the real and synthesized stimuli,
typing paper and aluminum foil were frequently confused, while copy paper and polypropylene were identified without such confusion. Further, the order of thecorrect answer ratios for the four materials was thesame for the real and the synthesized stimuli. Considering the collective experimental results, we conclude thatthe developed vibrotactile method captures the characteristic features of individual materials and the relationships between them. The method for displaying thecrinkle of a fragile structure is expected to provide virtual reality systems with immersive sensory feedback.

References
1. Agus M, Giachetti A, Gobbetti E, Zanetti G, Zorcolo A(2003) Real-time haptic and visual simulation of bonedissection. Presence 12(1):110–122
2. Arbabtafti M, Moghaddam M, Nahvi A, Mahvash M,
Richardson B, Shirinzadeh B (2011) Physics-based haptic simulation of bone machining. IEEE Transactions on
Haptics 4(1):39–50
3. Bensmaia S, Hollins M (2005) Pacinian representationsof fine surface texture. Perception & Psychophysics
67(5):842–854
4. Bensmaia S, Hollins M, Yau J (2005) Vibrotactile intensity and frequency information in the pacinian system:
A psychophysical model. Perception & Psychophysics
67(5):828–841
5. Bensmaı̈a SJ, Hollins M (2000) Complex tactile waveform discrimination. Journal of the Acoustical Society ofAmerica 108:1236–1245
6. Chim J, Lau RWH, Leong HV, Si A (2003) Cyberwalk: A
web-based distributed virtual walkthrough environment.
IEEE Transactions on Multimedia 5(4):503–515
7. Cholewiak SA, Kim K, Tan HZ, Adelstein BD (2010)
A frequency-domain analysis of haptic gratings. IEEE
Transactions on Haptics 3(1):3–14
8. Fujino S, Sato D, Abe K, Konno A, Uchiyama M (2008)
Displaying feeling of cutting by a micro-scissors type haptic device. In: Proceedings the 2008 IEEE International
Conference on Robotics and Automation, pp 2067–2072
9. Israr A, Tan HZ (2006) Frequency and amplitude discrimination along the kinesthetic-cutaneous continuum
in the presence of masking stimuli. Journal of Accoustical Society of America 120(5):2789–2800
10. Kavounoudias A, Roll R, Roll JP (1999) Specific wholebody shifts induced by frequency-modulated vibrationsof human plantar soles. Neuroscience 266:181–184

Spectrum-Based Synthesis of Vibrotactile Stimuli: Active Footstep Display for Crinkle of Fragile Structures
11. Law AW, Peck BV, Visell Y, Kry PG, Cooperstock JR(2008) A multi-modal floor-space for experiencing material deformation underfoot in virtual reality. In: Proceedings of the IEEE International Workshop on Haptic
Audio Visual Environments and Games, pp 126–131
12. Lurie KL, Shull PB, Nesbitt KF, Cutkosky MR (2011)
Informing haptic feedback design for gait retraining. In:
Proceedings the 2011 IEEE World Haptics Conference,
pp 19–24
13. Maki BE, Perry SD, Norrie RG, McIlroy WE (1999)
Effect of facilitation of sensation from plantar footsurface boundaries on postural stabilization in young
and older adults. The Journals of Gerontology: Series A
54(6):M281–M287
14. Morioka M, Griffin MJ (2005) Perception thresholds forvertical vibration at the hand, seat and foot. In: Proceedings of the European Acoustic Association Forum
Acusticum 2005, pp 1577–1582
15. Okamoto S, Ishikawa S, Nagano H, Yamada Y (2011)
Spectrum-based vibrotactile footstep display for crinkleof fragile structures. In: Proceedings of the 2011 IEEE
International Conference on Robotics and Biomimetics
pp 2459–2464
16. Okamoto S, Yamada Y (2011) An Objective Index
that Substitutes for Subjective Quality of Vibrotactile Material-Like Textures. In: Proceedings of the2011 IEEE/RSJ International Conference on Intelligent
Robots and Systems, pp 3060–3067
17. Okamura AM, Webster III RJ, Nolin JT, Johnson KW,
Jafry H (2003) The haptic scissors: Cutting in virtualenvironments. In: Proceedings of the 2003 IEEE International Conference on Robotics and Automation, pp 828–
833
18. Park G, Choi S (2011) Perceptual space of amplitudemodulated vibrotactile stimuli. In: Proceedings of the2011 IEEE World Haptics Conference, pp 59–64
19. Rafferty MM, Aliaga DG, Popescu V, Lastra AA (1998)
Images for accelerating architectural walkthroughs. IEEE
Computer Graphics and Applications 18(6):38–45
20. Roll R, Kavounoudias A, Roll JP (2002) Cutaneous afferents from human plantar sole contribute to body postureawareness. Neuroreport 13(15):1957–1961
21. Velázquez R, Bazán O, Magaña M (2009) A shoeintegrated tactile display for directional navigation. In:
Proceedings the 2009 IEEE/RSJ International Conference on Robots and Inteligent Systems, pp 1235–1240
22. Visell Y, Cooperstock JR, Giordano BL, Franinovic K,
Law A, McAdams S, Jathal K, Fontana F (2008) A vibrotactile device for display of virtual ground materials
in walking. In: Proceedings of EuroHaptics 2008, pp 420–
426
23. Visell Y, Law A, Cooperstock JR (2009) Touch is everywhere: Floor surfaces as ambient haptic interfaces. IEEE
Transactions on Haptics 2(3):148–159
24. Wakamatsu H, Zhang X, Honma S (1998) Teleoperational
force display system in manipulation of virtual object
using scissors-type cutting device. In: Proceedings of the
3rd Asia-Pacific Conference on Control & Measurement,
pp 312–316
25. Wang D, Zhang Y, Wang Y, Lee Y, Lu P, Wang Y (2005)
Cutting on triangle mesh: Local model-based haptic display for dental preparation surgery simulation. IEEE
Transactions on Visualization and Computer Graphics
11(6):671–683
26. Watanabe J, Ando H, Maeda T (2005) Shoe-shaped interface for inducing a walking cycle. In: Proceedings of11

the 2005 International Conference on Augmented TeleExistence, pp 30–34
27. Wiertlewski M, Lozada J, Hayward V (2011) The spatial
spectrum of tangential skin displacement can encode tactual texture. IEEE Transactions on Robotics 27(3):461–
472