2013 Ninth International Conference on Intelligent Information Hiding and Multimedia Signal Processing

Personal Auditory Tele-existence System Using a TeleHead

Tatsuya Hirahara

Daisuke Morikawa

Yukio Iwaya

Faculty of Engineering
Toyama Prefectural University
Imizu, Japanhirahara@pu-toyama.ac.jp

School of Information Science
JAIST
Nomi, Japan
morikawa@jaist.ac.jp

Faculty of Engineering
Tohoku Gakuin University
Tagajo, Japan
iwaya.yukio@tjcc.tohoku-gakuin.ac.jp

world the locations of sound sources are given, but in the realworld they are not. And determining the locations of soundsources in the real world is not an easy task.
If we had a dummy head that had the same shape as thelistener's head and tracked the listener's head movement, it
could provide perfect dynamic binaural signals for thelistener. A TeleHead is a steerable personal dummy head thattracks a listener's head movement in real time quietly [2].
Two small microphones at its right and left ears provide
dynamic binaural signals associated with the listener's head
movement. Nothing is taken away from or added to theelectrical signals they produce. The signals represent theactual binaural sounds obtained physically without using
information about the locations of the sound sources. Whenthe binaural signals are reproduce by earphones at thelistener's ear, the listener perceives 3D sound as if he/shewere at the remote place. A TeleHead can be an ideal
auditory tele-existence device.
In this paper, we describe a personal auditory teleexistence system that uses a TeleHead over the Internet. We
also demonstrate its performance by showing the results ofsound localization experimentswith the system.

Abstract— A TeleHead is a steerable dummy head that tracks alistener's head movement quickly and quietly. We made apersonal auditory tele-existence system connecting Toyama
Prefectural University and Research Institute of ElectricalCommunication Tohoku University by using a TeleHead overthe Internet. The remote TeleHead can provide dynamic
binaural signals associated with the head-movement data input,
and the local listener can hear the sounds represented by them.
The listener perceives the three-dimensional sound image as if
he were at the remote place even when the system has a totallatency of 223 ms.
Tele-existence; TeleHead; Dynamic binaural signal; 3D
sound; Internet (key words)

I.

INTRODUCTION

Telephones enable us to communicate verbally over long
distances. They virtually put the speaker's mouth close to thelistener's ear by bringing to a remote receiver the electricalsignals transduced by a local microphone. Although a
traditional telephone uses monaural sound, recent
technologies enable us to handle three-dimensional (3D)
sound. One that is promising for realizing a personal 3D
sound communication system is binaural technology.
In the digital era, binaural signals are usually synthesizedwith a head-related transfer function (HRTF)-based digitalbinaural system. Most of the digital binaural systems,
however, take something away from the synthesized binauralsignal and add something to them. Some are static systems,
which do not handle dynamic binaural signals associatedwith a listener's head movement. And even the dynamic
binaural systems usually remove early reflections as well as
reverberations because head-related impulse responses(HRIRs) reduced in length are used to synthesize binauralsignals. Doppler effects caused by the listener's head
movement are also usually ignored. In a dynamic binauralsystem, the spatial resolution of HRIRs should be less than 1
degree in order to avoid impulsive noises due to waveform
discontinuities [1]. HRIRs of such fine spatial resolution are
not measured, but are obtained by interpolating the measured
HRIRs of coarse spatial resolution (e.g. 5 to 10 degrees). The
use of interpolated HRIRs of course makes the synthesizedbinaural signals different from the actual binaural signals. In
addition, any digital systems synthesizing binaural signals
should know the locations of the sound sources. In the virtual
978-0-7695-5120-3/13 $26.00 © 2013 IEEE
DOI 10.1109/IIH-MSP.2013.88

II.

ARCHITECTURE OF TELEHEAD

Figure 1 shows the architecture of TeleHead IV [3]. A
listener’s head movement is tracked by a motion sensor(Ascension Technology's Flock of Birds) fixed at the top of
the head. The head rotation angle data sampled at 120 Hz aresent to a servo amplifier via a control PC. TeleHead V has
the same architecture, but its motor and servo amplifier are
different from those of TeleHead IV, and its motion sensor is
Polhemus FASTRAK.
The dummy head is a personalized one accurately
reproducing the shape of an actual listener's head. In thework reported here, each listener was an adult male listener
and the dummy heads were made by using a rapid
prototyping system. The 3D shape of each listener's head
was measured by a 1.5 T magnetic resonance imaging (MRI)system.
A small electret condenser microphone (SONYECM77B) embedded in an earplug made of silicone
impression material was placed in the vicinity of each of theouter-ear canal entrances (left and right) of the dummy head.
The output signals of the microphones were amplified 35 dB
by microphone amplifiers (Earthworks 1021) and headphone322

amplifiers (audio-technica AT-HA20), and then reproducedwith closed-type circumaural dynamic headphones(Sennheiser HDA200) [4].
III.

Audio Lines
Microphone Amp.

Headphone Amp.

Motion Sensor

Flock of Bird

ControlComputer

Listener

TELEHEAD OVER THE INTERNET

To use the TeleHead over the Internet, one needs a twochannel wideband sound codec for binaural signals and aserial data communication device for head movement data
(Figure 2).

Dummy Head

A. Binaural-signal transmission device 1
One of the binaural signal transmission devices we used
was a video teleconference system (SONY, PCS-XG55) witha built-in an MPEG-4 advanced audio encoding (AAC)
stereo codec (192 kbps). When the two video conference
systems were connected to the local area network (LAN) of
Toyama Prefectural University (TPU), the latency of thebinaural signals transmission was 370 ms. The round-trip
packet delay between the two systems measured by the pingcommand was less than 1 ms. Thus, most of the 370 ms was
the encoding and decoding time in the AAC codec.
When the two video conference systems were connected
over the Internet between TPU and the Research Institute ofElectronic Communication (RIEC) Tohoku University, theround-trip latency of the binaural signals transmission
including encoding and decoding time at both side was 1268
ms. Thus, the one-way latency was 264 ms. On the other
hand, the one-way packet delay between the two systems atthat time was about 23 ms.
The interaural time difference (ITD) of the binauralsignals differed by as much as 20 s between input andoutput signals, and the corresponding interaural level
difference (ILD) was as much as 0.5 dB. The frequency
response of the system was ±2 dB between 200 Hz and 5
kHz but dropped to 6 dB at 100 Hz and 12 kHz. Thisdifference in frequency response must be due to the codec
used in the system.

Servo Amp.

Figure 1. Architecture of TeleHead

The one-way latency for transferring the binaural signals
from the RIEC to TPU was 80 ms with the low-latency
codec system. The one-way packet delay measured by theping command was also about 23 ms at that time. There was
no ITD difference between input and output binaural signals,
but the ILD difference between them was 1.9 dB. The originof this ILD difference must be the difference of gain
between the right- and left-channel analog amplifiers in thecodec system. The frequency response of the system wasflat form 100 Hz to 4 kHz but dropped to -6 dB at 20 kHz.
C. Head-movement-data transmission device
Head-rotation-angle data of a listener measured every
1/120 seconds by the head tracker were sent to the TeleHead
controller via an RS-232C serial interface. The local and theremote RS-232C lines were connected over the Internet via
UDS2100s device servers (Lantronix).
When the listener's head-movement data were loopbacked RIEC-TPU-RIEC over the Internet with TeleHead V,
it took about 145 ms to reproduce the listener's head
movement. As the mechanical motion delay of TeleHead V
is about 100 ms, the round-trip latency of the data transfer
was estimated to be 45 ms, which was also the round- trip
latency for a single packet.

B. Binaural-signal transmission device 2
Another binaural signal transmission device we used
was a low-latency stereo-sound codec system (APT,WorldCast Astral), which offered high-quality audio
transport over IP networks using RTP/UDP. From several
types of sound codec available with the codec system, wechose the 20-kHz bandwidth 16-bit Linear PCM codec.

Dynamic binaural signals

TPU

LAN

䅇

RIEC

LAN

䅇

DS

motor

X

codec

䅇

SW
hub
PC

DS

X

FOB

SW
hub
codec

䅇

the Internet

Listener

TeleHead-IV
Head movement data
Figure 2. Tele-existence system connecting a TeleHead IV in Imizu with a listener in Sendai.

323

which of the 12 angles it came from as well as whether or
not the sound image was in-head.
In the first experiment, conducted on July 2011, twovideo conference systems were used to connect the
TeleHead IV at TPU and the listener at the RIEC. Threemale listeners participated in the experiment. The dummy
head put on the TeleHead was not their own personalized
dummy head but someone else's. In the second experiment,
conducted on March 2012, two low-latency sound codec
systems were used to connect the TeleHead IV at TPU andthe listener at the RIEC. Three other male listeners
participated in the experiment. The dummy head used waseach listener's own.
In the third experiment, conducted on March 2012, theTeleHead IV and the listener were both at TPU and wereconnected directly by wires. The total latency of the system
was 120 ms. The listeners in this experiment were the same
three listeners who participated in the second experiment.
The dummy head used was each listener's own.

D. Total transmission latency over the Internet
When the video conference systems were used, the timerequired for the binaural signals received by two
microphones in a remote TeleHead being reproduced at a
local listener's ears was 634 ms, which was the sum of 370
ms for coding and decoding and 264 ms for the binauralsignal-data transmission over the Internet. The time required
for the local listener's head movement being reproduced at
the remote TeleHead was 143 ms, which was the sum of 23
ms for the data transmission over the Internet and 120 ms
for the mechanical motion delay of the TeleHead-IV. It
therefore took 777 ms for the listener to hear the dynamic
binaural sounds provided by the remote TeleHead, in which
his head movement was reflected.
When the low-latency sound codec systems were used,
the latency for the binaural signals was only 80 ms, which
included the time for coding, the data transmission over theInternet, and the time for decoding. Therefore the totallatency of the system was 223 ms.
IV.

B. Results
Figure 3 shows the pooled results for the three listeners
in the head-still and head movement conditions in the firstexperiment. In each panel the area of the blue-filled circles
is proportional to the in-head localization rate, and that of
the red-filled circles is proportional to the out-of-headlocalization rate.
In the head-still condition most stimuli were localized
out-of-head but some presented at a front position werelocalized in-head. There were also some front-backconfusions and near-miss orientation-judgment errors. As aresult, the mean correct out-of-head localization rate Pc was
45%. In the head-movement condition, there were fewer inhead localizations and front-back confusions than there werein the head-still condition. Near-miss localization errors,
however, remained, resulting in a Pc of 51%. All of thelisteners reported that it took so long for the head movement
to be reflected in the sound-image movement that soundlocalization was not easy in the head-movement condition.
Figure 4 shows the pooled result for the three listeners
in the head-movement conditions in the second experimentin which the head-still condition was skipped. Since it was
the head-movement condition, there were fewer in-headlocalizations, front-back confusions, and near-missorientation-judgment errors, resulting in a Pc of 82%. All of
the listeners reported that they could localize each stimulus
easily even though the delay between the head movement
and sound-image movement was clearly perceptible.
Figure 5 shows the pooled results for the three listeners
in the head-still and head-movement conditions in the third
experiment. In the head-still condition, most stimuli werelocalized out-of-head but some presented at the front
position were localized in-head. Some front-back confusions
and near-miss orientation-judgment errors occurred. As aresult, the mean correct out-of-head localization rate Pc in
the head-still condition was 84%. In the head-movement

PERFORMANCE EVALUATION

The performance of the tele-existence system using
TeleHead IV was evaluated by conducting horizontal soundlocalization experiments.
A. Procedure
The TeleHead IV was placed in an experimental room at
TPU and 12 loudspeakers were placed around it in thehorizontal plane at intervals of 30°. The distance from theloudspeakers to the center of the dummy head was 1 m. The
A-weighted noise level of the room was 22 dB and thereverberation time was about 50 ms.
White noise was used as the sound stimulus. It was
generated on a PC with a sampling frequency at 48 kHz,D/A converted by USB audio interface units (Roland UA101), amplified by an amplifier (Bose 1705 II), and fed to a
loudspeaker. The binaural signals captured by the dummy
head at TPU were transferred over the Internet toheadphones worn by a listener sitting in a sound-proof roomat the RIEC. The listener at the RIEC thus listened to thesound stimuli through the remote dummy head’s ears, as if
he sat at the center of the speaker array in the experimentroom at TPU.
The duration of each stimulus was 3 s and the interstimulus interval was 5 s. The sound pressure level of thestimulus presented from the front speaker was set at 70 dB,
and the stimulus reproduced by headphonesat the RIEC
was presented to the listener at 70 dB.
Sound localization experiments for the head-still andhead-movement conditions were conducted separately. Eachexperiment consisted of four sessions, and every session
consisted of 60 trials in which white noise was randomly
presented five times from each of the 12 azimuthal angles.
Twenty responses were thus obtained to sound from each of
the 12 angles. Listeners were asked to judge from where thesound was presented and to indicate, on an answer sheet,324Perceived azimuth angle [ 㫦]condition, the Pc was 88%. Although in the head-movement
condition there were no in-head localizations and there werefewer front-back confusions than there were in the head-still
condition, near-miss localization errors remained.
C. Discussion
Even when the total latency of the system was 777 ms,
and the listener used someone else's dummy head, front-backconfusions were diminished by the use of dynamic binauralsignals provided by the remote TeleHead. The correct out-ofhead sound localization rate Pc in the head-movement
condition, however, was only 51%. This relatively low value
is probably due to the long system latency. The results of thefirst experiment suggest that the impact of the head
movement in sound localization [5, 6] is reduced when thesystem latency is too long. This might be because then thelistener's brain cannot associate the listener's head movementwith movement of the sound image [3].
When the total latency of the system was 223 ms and thelistener used his personalized dummy head, the Pc in thehead-movement condition was 85%. This rate was almost thesame as that obtained using a local TeleHead IV. The use of
the dynamic binaural signals overcame the long system
latency. The results of the second experiment demonstratethat a TeleHead used with a low-latency codec system works
over the Internet as a personal auditory tele-existence systemwith satisfactory performance.

Head䋭still

Head䋭movement240
180
in head120
60

out-of head

0
120

360 0

240

120

240

360

Target azimuth angle [ 㫦]Perceived azimuth angle [ 㫦]Figure 3. Results of the first experiment with a remote TeleHead andlocal listener connected by a video conference system.
360
300

Head䋭movement

240
180
120
60
0
0

120

240

360

Target azimuth angle [ 㫦]Figure 4. Results of the second experiment with a remote TeleHead andlocal listener connected by a low-latency audio codec system.

CONCLUSION

A personal auditory tele-existence system between TPU
and the RIEC was implemented over the Internet by using a
low-latency codec to connect the remote TeleHead IV andthe local listener. The total delay time between the listener's
head movement and the corresponding sound image
movement was 223 ms: 23 ms for sending the head
movement data through an RS-232C port, 80 ms for sending
the binaural signals, and 120 ms for the mechanical motion
delay of TeleHead IV. The mean correct-out-of-head soundlocalization rate for the horizontal sound sources in the headmovement condition was 85%, which was comparable withthat obtained using a local TeleHead IV. The total system
latency with the video conference system was about 780 ms,
yielding poorer performance. When the system latency is too
long, the listener's voluntary head movement is less effective
in sound localization.

360
300

Head䋭still

Head䋭movement

240
180
120
60
0
0

120

240
360 0
120
Target azimuth angle [ 㫦]

240

360

Figure 5. Results of the third experiment with a TeleHead at TPU andlistener at TPU connected directly by wire.
[2]

ACKNOWLEDGMENTS[3]

Part of this work was carried out under the Cooperative
Research Project Program of the RIEC, Tohoku University.
Authors would like to thank Professor Yôiti Suzuki forsupporting the project. Part of this work was also supported
by JSPS KAKENHI Grant Number 22300061.

[4]

[5]

REFERENCES
[1]

300

0

Perceived azimuth angle [ 㫦]

V.

360

M. Otani, T. Hirahara, “Auditory Artifacts due to Switching HeadRelated Transfer Functions of a Dynamic Virtual Auditory Display,”
IEICE Trnas. on Fundamentals of Electronics, Communications andComputer Sciences E91-A No.6, pp.1320-1328 (2008).

[6]

325

I. Toshima㧘S. Aoki㧘T. Hirahara㧘“Sound localization using anauditory telepresence robot: TeleHead II,” Presence, Vol. 17, No. 4,
pp. 392-404, (2008).
T. Hirahara, D. Yoshisaki, D. Morikawa, “Impact of dynamic binauralsignal associated with listener's voluntary movement in auditory spatial
perception,” Proceedings of Meetings on Acoustics, Vol. 19, 050130,
pp. 1-8 (2013).
T Hirahara, “Physical characteristics of headphones used in
psychophysical experiments,” Acoustical Science and Technology Vol.
25, No.4, pp. 276-285 (2004)
D. Morikawa, T. Hirahara, “Effect of head rotation on horizontal andmedian sound localization of band-limited noise,” Acoustical Scienceand Technology Vol.34, No.1, pp.56-58, (2013)
Y. Iwaya, Y. Suzuki, D. Kimura, “Effects of head movement on frontback error in sound localization,” Acoustical Science and Technology
Vol. 24, No. 5, pp. 322–324 (2003).