Real-time Contact Sound Synthesis for Multisensory Interaction
JUAN LIU† ††, and HIROSHI ANDO† ††1. Introduction

3. Integrating Sound Models and User Actions

The sense of hearing plays many important roles in ourdaily lives. We may produce abundant sounds by interactingwith objects in our reach. These sounds convey plenty ofinformation about the object, such as material, roughness,stiffness, the speed and type of our actions et al. Concurrent
sound feedback provides us a sense of presence and realism. In
virtual reality systems, users may immediately notice theunnatualness if the interface has no sound effect or provides
mismatched sound.
This paper addresses the problem of real-time audio
synthesis for haptic interface and provides a physicallymotivated integrated model for various contact interactions,such as impacting, scratching and inhibiting. The prototype
system based on a force feedback device (PHANTOM®) isable to offer tightly synchronized visual-haptic-auditory
stimuli, which makes users feel like manipulating the real
object.

Contact sounds we hear during our interaction with solid
objects are waves radiated by vibrating structures. Various
materials and shapes deform in different ways so that we cantell the difference of their sounds. To simulate the interactive
sounds emitted from a solid object, we may use a physicallymotivated modal synthesis model M = {f , d, A} , which
describes the object by a bank of damped harmonic oscillatorswith modal frequencies f n , decay rates d n and amplitudes

2. Related WorkFor continuously interactive sound synthesis, playing backprerecorded sounds is unsatisfactory and inapplicable since wecannot predict the changes in the sound. People excite soundproducing objects with a variety of other objects in a variety ofways. A lot of researches have been carried out in acoustics,physics and digital signal processing for physical model ofsound production and parameterized sound synthesis methods[1]. Van den Doel, Kry and Pai [2] mapped analyticallycomputed vibration modes onto object surfaces forautomatically synthesizing sound effects for animation andsimulation. However, they used different methods for impact,
scraping and rolling scenes and the objects they simulated aremade of one kind of material and simple structure. Because inanimation the transition among motions can be predicted fromthe dynamic model, it is unclear whether their models canwork well for human-machine interaction, in which users’
intention and active action introduce new problems. O’Brien,
Cook and Essl [3] described an off-line system to compute
sound and motion from a single physical model of deformablebodies. The sound is generated by computing the propagationof acoustic pressure waves induced by motion. Barrass andAdcock [4] implemented scraping sound using granularsynthesis for haptic workbench. But the impact sound was not
integrated effectively in their system. Since our aim is tosynthesize realistic sound for arbitrary contact actions carriedout through haptic interface, our work is closely related to online algorithms based on a physical model.
†Multimodal Communication Group, Universal Media ResearchCentre, National Institute of Information and Communications
Technology (NiCT)†† Cognitive Information Science Laboratories, Advanced
Telecommunications Research Institute International (ATR)

an where n = 1,..., N [1, 2]. The vibrating structure is taken
as a linear time-invariant system (LTI), which can becharacterized by its impulse response h(t ) .
Nh(t ) = ∑ an e−dn t sin(2π f n t )

(1)

n=1

Contact sound y (t ) , the output of the system, is theconvolution of the input stimulus x(t ) and the impulse
response, i.e. y (t ) = ( h ∗ x)(t ) . Therefore the sound synthesis
problem can be decomposed into two parts, i.e. to obtain
vibration model parameters and to get input excitation model.
In many situations if the way of interaction is different,such as striking or scraping, contact sounds are totallydifferent even when we touch at the same location. Therefore
several models ( i.e. several sets of parameters) are necessary
in order to provide a more complete description of the acoustic
features of one surface. Then how should we trigger thesemodels to generate sound? It is intuitive to choose a top-down
strategy that excites corresponding model according to thetype of an action (impacting or scratching). We call it thebranched method. It might work for sound effect synthesis foranimation with predefined plot, but in real-time interaction, theintention of the user is blind to the system. The system has tojudge what kind of action is carrying out. However, to gather
certain amount of data for the judgment takes time, while thesound has to be generated at the same moment when the user’s
stylus touches the surface. Since human’s auditory temporal
resolution is much higher than that of somatic sensation, in asystem that updates haptic output at 1kHz and generates
44.1kHz audio output, the latency of judging process can benoticeable and annoying. Meanwhile, the fluctuation at theboundary of actions and sound models will also make thesynthesized sound noise and unnatural.
We argue that a bottom-up strategy is more suitable. If we
put the user’s intention aside, and just analyze the vibration of
the surface, in a process of interaction, the way of vibration is
not pure impact or scratching mode. For example, when thestylus touches the surface, normal force will trigger the modesof vibration in impact model, even the user intend to scrape thesurface. Or in some cases, the impact behavior may also
introduce tangential element that excites the scratching soundmodel. These models are superposed rather than exclusive. So
we propose a physically motivated way to integrate theexcitation of sound models. An inhibition factor is also
introduced to characterize a typical interactive action thatpeople intent to attenuate the sound by pushing the surface.
Take two kinds of interactive actions and sound models as
an example (impact and scratching). Assume there is only one
contact point at one moment. The synthesized sound is:y (t ) = (himp ∗ ximp )(t ) + (hscr ∗ xscr )(t )(2)where the input stimuli ximp and xscr are functions ofsquare algorithm. Finally each sound model is associated withthe part of the object where their samples are taken. The datafrom the haptic device are used to control the output of soundmodels as described in Section 3.
Users wearing a stereo shutter glasses may see the 3D
image of the object, touch it with the stylus in various ways,
feel the force-feedback and simultaneously hear the contact
sounds of high fidelity.

contacting time, normal force exerted on the surface Fn andtangential force Ft . Along the increase of contact duration,
the possibility of the action being an impact decreases,
meanwhile that of the scratching action increases. The ratio ofsounds changes accordingly.

ximp = wimp (t )(1− ρ ) Fn (t ),xscr = wscr (t )(1− ρ ) Ft (t ) ,wimp = e−α (t −t0 )φ , wscr = 1− e−α (t −t0 )φ .
φ = {0,1} is used to indicate whether the proxy of the stylus
is touching the surface or not.
0: not contacting; 1: contacting the surface since t0 .
α is a coefficient which is related to the material.
ρ = {0,1} is used to indicate whether the contact is aninhibition action or not. The change of position of the proxy on
the surface in consecutive 2 cycles (totally 2 ms) is taken as acriterion.
0: more than or equal to threshold δ , no inhibition;
1: less than threshold δ , taken it as an inhibition behavior.
The inhibition action dramatically damps the sound:N−( ρ Dimp +1) dhimp = ∑ a imp
enimptnn =1
N−( ρ Dscr +1) d scr thscr = ∑ a nscr en =1nsin(2π f nimp t ) ,sin(2π f nscr t ) ,where Dimp > 1 , Dscr > 1 , related to the material and thecontact area of the stylus and the surface.

Fig. 1 Multi-sensory interaction system showing Japanesedrum demonstration

5. Conclusions and Future WorkA method to utilize consistent sound synthesis models forvarious contact interactions is proposed and implemented in a
multi-sensory interaction system. Since we can precisely
control the frequency component, damping and amplitude
parameters as well as the excitation of sound models withhaptic feedback, we are going to devise some psychophysicalexperiments applying this system to investigate auditory cuesthat influence people’s perceptual judgments and theirrelationship with other modalities. The next version of thesystem may incorporate spatial information of the soundsources and radiation efficiency. We are also working on
aerodynamic sound and liquid sound to provide a wider
application prospect for the system in design, entertainment,education, training and telecommunication.

4. Manipulating Objects: from Real to Virtual
The purpose of our work is to duplicate a real object in thevirtual environment, and when people interact with the virtualobject, they may have the feeling that the real one is present infront of them. That requires us to provide multi-sensory
stimuli and synchronize them closely. The prototype system asshown in Fig.1 consists of Reachin® 3D display, PHANTOM
force-feedback device and the integrated audio part utilizing
our real-time sound synthesis method proposed in above
section.
Firstly the 3D shapes and texture images of the object arecaptured by a non-contact 3D digitizer (Vivid 910 from
Konica Minolta), and various contact sounds of the real object
are recorded in an anechoic room. The shapes and images
taken from different viewpoints are stitched together forming acomplete 3D model for visual and haptic rendering. Theparameters of contact sound models are obtained by fitting
them to the sampled sound files. For each sound model,
around 200 mode frequencies are selected from the result ofwindowed discrete Fourier transformation. Then the damping
and coupling amplitude parameters are estimated using least

References
1) Cook, P. R.: Real Sound Synthesis for Interactive
Applications, A K Peters, Wellesley, MA (2002)2) van den Doel, K., Kry, P. G., and Pai, Dinesh K.:
FoleyAutomatic: Physically-based Sound Effects forInteractive Simulation and Animation, in Computer Graphics(ACM SIGGRAPH 01 Conference Proceedings), pp. 537-544(2001).
3) O'Brien, J. F., Cook, P. R., and Essl, G.: Synthesizing
Sounds from Physically Based Motion, in Proceedings of
SIGGRAPH 2001, Annual Conference Series, (Los Angeles,California), pp. 529-536 (2001)
4) Barrass, S., Adcock, M.: Interactive Granular Synthesis of
Haptic Contact Sounds, in Proceedings of the AudioEngineering Society 22nd International Conference on Virtual,
Synthetic and Entertainment Audio (AES22), Espoo, Finland.
AES. pp. 270-277 (2002)