MqMqMqPrevious Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page qMqM
THE WORLD’S NEWSSTAND®

Interactive Sonification

Wearable
Auditory
Biofeedback
Device for
Blind andSightedIndividuals
Masaki Matsubara, Takahiro Oba, Hideki Kadone,
Hiroko Terasawa, and Kenji Suzuki
University of Tsukuba
Masaki Iguchi
Tsukuba University of Technology

GaitEcho is a
wearable auditory
biofeedback device
that uses aninstrumented anklefoot orthosis for gait
rehabilitation.
Experimental results
suggest it offers
similarly adequate
functionality for blindand sighted
participants.

68A

s a population ages, elderly people
increasingly fall and injure themselves. Such falls make them hesitant to move actively and thus
severely decrease their independence and quality of life. Preventing falls is therefore critical inmaintaining the wellness of elderly people.1
In physiotherapy, a tracking task for anankle-joint exercise can be used to facilitate
motor learning of the ankle.2 For such tasks, aphysiotherapist teaches a movement patternthat is optimally designed for the patient (reference), and the patient practices the pattern(tracking). However, Masaki Iguchi, our physiotherapist researcher (the second author) suggeststhat in practice such tasks tend to bore patients,
who as a result may discontinue rehabilitation.

!

1070-986X/15/$31.00 c 2015 IEEE

Making the task effective, yet enjoyable would
therefore encourage better exercise participation, resulting in fall prevention over the longterm.
Biofeedback has been successfully used inphysical rehabilitation for approximately 40
years,3 but visual biofeedback (VBF) examples
greatly outnumber auditory biofeedback (ABF)
examples.4–6 Earlier work examined the effectiveness of VBF in a tracking task using realtime visualization of ankle-joint movement,2
however, the effectiveness of ABF in an anklejoint tracking task has not been investigated.
Because ABF does not require visual attention,
it provides a much broader possible range of
postures and gestures than VBF. Unattended
use of vision is beneficial in physical exercise
such as gait training.7 Therefore ABF can overcome some drawbacks of VBF in rehabilitation.
In addition to such physical advantages, ABF
can widen the user population because visually
impaired and blind individuals can benefit
greatly from such a system.6 Furthermore, sincemusic can encourage accurate motor activities
with positive emotional effects in physical therapy,8 we expect similar effects by introducing asonic interaction with rhythmic and melodic
aspects. To address this issue, we developed awearable auditory-biofeedback device called
GaitEcho that can be used in fall prevention
programs for sighted and blind individuals andexamined the effects of interactive sonification
on an ankle-joint exercise.
This study discusses the feasibility of a reference-tracking task of ankle-joint exercise withinteractive sonification for sighted and blind
individuals in terms of both objective and subjective evaluations via our wearable auditory
biofeedback device using the instrumented
ankle-foot orthosis (AFO) we call GaitEcho. Our
experimental results suggest it offers similarly
adequate functionality for both blind andsighted participants.

System Configuration
GaitEcho measures the instantaneous angle atthe ankle joint (the ankle-angle signal) with agoniometer (Supertech Electronic P-00246)
mounted on a custom-made plastic AFO (Kowagishi Laboratory) and sends it to a tablet PC(Acer Iconia W3-810) with Bluetooth serial
communication at 500 Hz. To add sonification,a server program on the PC smoothes thereceived ankle-angle signal with a movingaverage filter and sends it to a sound synthesis

Published by the IEEE Computer Society
M
q
M
q

M
q

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page qMqM
THE WORLD’S NEWSSTAND®

M
q
M
q

M
q

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page qMqM
THE WORLD’S NEWSSTAND®

Figure 1. Overview of
GaitEcho. The

Headphone

wearable auditory
biofeedback device
uses an instrumented
ankle-foot orthosis for

Audio

gait rehabilitation.

Wearable
control

Battery
LED display
Angle sensor

Ground contact sensor

actively lengthened, with the decreasing pitch
analogous to decreasing tension.
A finger-snapping sound was added to thecontinuously sweeping sinusoid so that a patient
can notice when the required movement isaccomplished. This sound is generated when theangle at the ankle joint reaches the maximum
dorsiflexion and plantarflexion angles.
We implemented this sonification using
SuperCollider. (Supplementary files and sourcecode are available in an online appendix at
https://db.tt/qwvSdGxe.)_________________

Experiments
Six sighted participants (five males and one
female, aged 22 to 31) and six blind participants(four males and two females, aged 20 to 25) participated in our experiments. All participants
gave their informed consent to the experimental procedure, which was approved by the local
ethics committee. Sighted participants performed the task with both ABF and VBF andblind participants with ABF only.

January–March 2015

server with Open Sound Control protocol(at 25 Hz). A reference movement pattern is prerecorded and stored on the synthesis system.
The sound generated by the reference emanatesfrom the right channel of a headphone (SonyMDR-CD780), and the reference generated by
the ankle angle emanates from the left channel.
Figures 1 and 2 show the overview and specifications of GaitEcho.
Previous sonification research on continuous
human movement in rehabilitation showedthat continuous mapping approaches are suitable for motor learning.9–12 We employed a
mixture of continuous and discrete mapping: asinusoid whose frequency is swept according to
the ankle angle is mixed with a finger-snap
sound to indicate the designated maximum andminimum peak angles.
We determined the sinusoid frequency was
500 Hz at maximum dorsiflexion (toe-pointing
upward, ankle-bending direction) and 250 Hz
at maximum plantarflexion (toe-pointing
downward, ankle-extended direction). In dorsiflexion, the muscles at the shin (such as thetibialis anterior) are contracted with increasing
pitch that is meant to be analogous to increasing tension. In plantarflexion, the muscles are

Tracking Task and Its Evaluation
In the tracking task, participants performed amovement pattern as closely as possible to the

69
M
q
M
q

M
q

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page qMqM
THE WORLD’S NEWSSTAND®

M
q
M
q

M
q

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page qMqM
THE WORLD’S NEWSSTAND®

Figure 2. GaitEcho
system scheme.
GaitEcho sends the

Sensor input

Feedback

ankle-angle signal to aserver program that inturn forwards it to asound synthesis server
to generate a variable-

Goniometerfrequency sinusoidthat reflects the ankleangle and a clicksound when the ankle

Closedloop
BluetoothWearable control unit

joint reaches the
maximum
dorsiflexion andplantarflexion angles.
Mixing these two

Signalprocessing

Reference
movements
recordedby PT

Headphonesounds helps the user
simultaneously
understand both thecurrent ankle-angle

Parameter
mappingParameter
mapping

and the target angle.

Sound synthesis,
line out
Channel 1

Channel 2IEEE MultiMedia

reference using ABF or VBF. For the reference, aphysiotherapist (PT) moved one participant’s
ankle with AFO to record six movement patternsof 60 to 70 seconds. Each pattern comprised a
combination of four to seven movements withdifferent speeds. Each movement was about 25
degrees of dorsiflexion and 30 degrees of plantarflexion, taking 6 to 10 s. The dorsiflexion
angle was adjusted for some participantsbecause of their limited range of motion.
The evaluation of our system has some
exploratory character. Because the frameworkof the reference-tracking task requires a feedback, we cannot test the “no feedback” control
condition. However, we aimed for a comparison with the standard system practically tested
in the medical research, instead of a purely theoretical comparison as described in the discussion section.
Procedure of the Experiment
The participants first performed a practice task
and then the main performance task. The latter
took about 10 minutes for blind participantsand 20 minutes for sighted participants. The
order of the movements and that of the two
conditions (ABF and VBF) were both randomized for each participant.
Throughout the practice and main tasks,
participants sat up straight in a chair with theirleg slightly raised on a stool and performed theankle-joint tracking task in that sitting position. We recorded the entire waveform of theankle-angle signal and used it for the analysis.
After finishing the main task, we asked participants to answer a questionnaire.
For the practice task, the participants performed up to 10 minutes of free movementwith the visual/sonic representations until theyfelt confident, and they then performed a sample reference-tracking task. Blind participants
performed with ABF only, whereas sighted participants performed with both ABF and VBF.
For VBF, the reference was plotted as a waveform on a computer screen, replicating the system developed by Monica Perez, Jesper LundbyeJensen, and Jens Nielsen.2 Participants traced
the reference waveform with their own waveform, using a cursor-point that moves automatically from left to right and changes its vertical
position according to the participant’s ankle
angle. The cursor moved upward during dorsiflexion and downward during plantarflexion.
For ABF, the sonification of the reference
emanated from the headphone’s left side andthat of the participant’s movement emanatedfrom the right. Participants traced the reference
movement by synchronizing two sounds. The
sound design was basically identical for bothsounds, but to facilitate auditory segregation,
the reference sonification used pulse waves
instead of sinusoids. Participants could therefore easily distinguish reference movementsfrom their own by hearing left/right sounds.
The delay time between sensor measurement
and sonification was less than 0.04 s.
After the main task, participants rated on ascale from one to five three questionnaire
items:!

subjective understandability (Was the sonification hard/easy to understand?),!

task difficulty (Was the motor task difficult/easy?), and!

enjoyment
boring?).

(Was

the

task

enjoyable/

Finally, they gave free comments.

70
M
q
M
q

M
q

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page qMqM
THE WORLD’S NEWSSTAND®

M
q
M
q

M
q

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page qMqM
THE WORLD’S NEWSSTAND®

12
Average error on timing (%)

Average error on angle (%)

5
4
3
n.s.
2
1
0

VBF

ABF (sighted)(a)Subjective rating

the average error on

8(a) angle and (b)timing in tracking

6tasks with VBF andABF (sighted) andABF (blind) (n ¼ 6).

4
2
VBF

ABF (sighted)

ABF (blind)

(b)

5

*
5

*

n.s.
5

4

4

4
3

3

3

2

2
1

Figure 3. Average andstandard deviation of

100

ABF (blind)

n.s.

VBF

ABF (sighted)

ABF (blind)

(a)

1
(b)

2
VBF

ABF (sighted)

ABF (blind)1

VBF

ABF (sighted) ABF (blind)(c)Figure 4. Average and standard deviation of subjective (a) understandability, (b) difficulty, and (c) enjoyment in tracking tasks with
VBF and ABF (sighted) and ABF (blind) (n ¼ 6). For the ratings, 1 indicated hard to understand, easy to perform, or boring; 3 denoted
ordinary; and 5 indicated easy to understand, difficult to perform, or enjoyable, for the respective questions on understandability,
difficulty, and enjoyment.

Results
The error between the reference and participants’ movements was calculated using the differences of angle and timing at maximum
dorsiflexion and plantarflexion. For each movement, we obtained average error on angle (AEA)
and average error on timing (AET):
AEA ¼

!
!
N !
ari # asi !
100 X
;
A
N i¼1

where ari and asi are the reference and participants’ angles at the ith peak ði ¼ 1…NÞ, respectively, and A is the physiotherapist-instructed
range of the ankle-joint angle in the task.
!
!
N !
tri # tsi !
100 X
;
T
N i¼1

where tri and tsi are the reference and participants’ timing at the ith peak ði ¼ 1…NÞ, respectively, and T is the reference-instructed
duration of a movement.

January–March 2015

AET ¼

Figure 3 shows the mean and standard deviation of the AEA and AET. AEA showed no significant difference between ABF (sighted) and ABF(blind) (p ¼ 0:2732, unpaired student’s t-test).
The maximum AEA was less than 2 percent,equivalent to less than 1 degree of movement.
AET showed no significant difference between
ABF (sighted) and ABF (blind) (p ¼ 0:3228,
unpaired student’s t-test). The mean AET with
VBF was 0.30 s, and that with ABF (sighted andblind) was 0.66 s. The duration of each movement in the task exceeded 6 s, so these errors
comprised less than about 10 percent of the total
duration.
As Figure 4 shows, under the ABF condition,
blind participants gave higher understandability and enjoyment scores (p ¼ 0:0428 and
0:0380, respectively) than sighted participants,
while their difficulty scores (p ¼ 0:7805) were
not significantly different. Both sighted andblind participants commented that for bothconditions they enjoyed the tasks, comparing
the experience to playing a game.

71
M
q
M
q

M
q

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page qMqM
THE WORLD’S NEWSSTAND®

M
q
M
q

M
q

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page qMqM
THE WORLD’S NEWSSTAND®

Most blind participants
expressed delightthrough facial expression
when using GaitEcho,
while some reported that
their foot became like amusical instrument.

Discussion

IEEE MultiMedia

Our physiotherapist determined that bothsighted and blind participants appropriately
performed the ankle-joint exercise under both
ABF and VBF conditions. The AEA with ABF wasonly 1 degree, within the acceptable error
range. The 0.66 s AET with ABF is also acceptable because the ankle-joint exercise does not
usually require precise timing.
In physical rehabilitation, training methods
and rehabilitation programs are personallydesigned for the patient’s needs. If an exercise
requires precise timing, rhythm-based sonification can be used instead of pitch-modulation
sonification. The relationship between movement types in various context and sonification
methods has been investigated in the field of
sonification for locomotion.12
The difference between VBF and ABF insighted participants, especially for timing controllability, comes about because the systemdesign employs different kinds of tasks for ABF
and VBF. The VBF shows the entire reference
waveform and thus enables the participant topredict the subsequent movement, while the
ABF provides only the instantaneous ankle
angle. In our prototype, we tried using VBF thatshows the instantaneous ankle angle only,
and in that case, the timing error was actuallybigger than that with ABF. However, in thisstudy, we employed the current VBF design(proposed earlier2) as a standard representativefor visual biofeedback tested in the medical
research.
The questionnaire scores showed significant
differences between blind and sighted participants. Blind participants gave higher enjoy-

ment and understandability scores, while thetask performance difficulty ratings were similar.
During the experiment, most blind participants expressed delight through facial expression when using GaitEcho, while some blindparticipants reported that their foot became
like a musical instrument. For angle andtiming controllability, no significant difference
between the participants was seen with ABF.
This implies that ABF offers equivalent functionality for blind and sighted participants, yetprovides better emotional effect for blind
people.
Some sighted participants commented theyfelt more comfortable with ABF than VBF
because they felt no eyestrain with the former
while watching the display. ABF reduces eye
fatigue and allows for more varied position andposture. Some participants commented theyfelt no physical fatigue under either condition,
but felt mental fatigue with VBF.

Conclusion
With ABF, both the sighted and blind groups
performed the exercise appropriately, andblind participants reported higher understandability and enjoyment. Yet, the future applications are to be explored. GaitEcho is a fully
equipped wearable system, and people can use
it wherever they want, outside the typicalrehabilitation venues. GaitEcho would besuitable for bedside rehabilitations or everyday
rehabilitation at home. We also plan to apply
GaitEcho for gait training at home, or anywhere outside hospitals. Exploring emotionallyengaging sound design remains our future
problem, such as employing more musicalframework.
Demo video, sound files, supplementary
data, and source code for sonification (SuperCollider) associated with this article are available online at https://db.tt/qwvSdGxe.
MM________________

References
1. G.C. Gauchard et al., “Beneficial Effect of Proprioceptive Physical Activities on Balance Control inElderly Human Subjects,” Neuroscience Letters, vol.
273, 1999, pp. 81–84.
2. M.A. Perez, J. Lundbye-Jensen, and J.B. Nielsen,“Changes in Corticospinal Drive to Spinal Motoneurones Following Visuo-Motor Skill Learning in
Humans,” J. Physiology, vol. 573, no. 3, 2006, pp.
843–855.
3. J.V. Basmajian et al., “Biofeedback Treatment ofFoot-Drop after Stroke Compared with Standard

72
M
q
M
q

M
q

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page qMqM
THE WORLD’S NEWSSTAND®

M
q
M
q

M
q

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page qMqM
THE WORLD’S NEWSSTAND®

Rehabilitation Technique: Effects on Voluntary
Control and Strength,” Archives of Physical
Medicine and Rehabilitation, vol. 56, no. 231,

from the University of Iowa. Contact him at ___iguchi@k.tsukuba-tech.ac.jp.
________________

1975.
4. H. Huang, S.L. Wolf, and J. He, “Recent Developments in Biofeedback for Neuromotor Rehabilitation,” J. Neuroengineering and Rehabilitation, vol.
3, 2006, pp. 1–11.
5. J. Blum, K. Lutz, and L. Jancke, “Coherence andPhase Locking of Intracerebral Activation During

Takahiro Ohba is a system engineer at NEC Corporation. His research interests include flying robot
technology and assistive and rehabilitation robotics.
Ohba has an ME in intelligent interaction technologies from the University of Tsukuba. Contact him attaka@ai.iit.tsukuba.ac.jp.
_______________

Visuo- and Audio-Motor Learning of Continuous
Tracking Movements,” Experimental Brain Research,
vol. 182, no. 1, 2007, pp. 59–69.
6. M. Iguchi et al., “How Is Auditory Biofeedback
Effective for Blind People?” Proc. 19th Int’l Conf.
Auditory Display, 2013, pp. 307–310.
7. A.E. Patla, “Understanding the Roles of Vision in
the Control of Human Locomotion,” Gait & Posture, vol. 5, 1997, pp. 54–69.
8. V. Dietz, T. Nef, and W.Z. Rymer, eds., Neurorehabilitatioin Technology, Springer-Verlag, 2012.
9. N. Schaffert, K. Mattes, and A. Effenberg, “ASound Design for Acoustic Feedback in Elite
Sports,” Auditory Display, Springer, 2010, pp.
143–165.
10. T. Großhauser et al., “Wearable Sensor-Based RealTime Sonification of Motion and Foot Pressure inDance Teaching and Training,” J. Audio Eng. Soc.,
vol. 60, nos. 7/8, 2012, pp. 580–589.
11. L. Chiari et al., “Audio-Biofeedback for Balance
Improvement: An Accelerometry-Based System,”IEEE Trans. Biomedical Eng., vol. 52, no. 12, 200,
pp. 2108–21115.
12. Y. Visell et al., “Sound Design and Perception in
Walking Interactions,” Int’l J. Human-Computer
Studies, vol. 67, 2009, pp. 947–959.

Masaki Matsubara is an assistant professor, co-principal investigator in the Laboratory for Sound andPeople with Computing, and a researcher in the
Research Center for Knowledge Communities atUniversity of Tsukuba. His research interests include
music technology, music cognition, and assistiveand rehabilitation technology. Matsubara has a PhD
in engineering from Keio University. Contact him at

Hideki Kadone is an assistant professor in theCenter for Innovative Medicine and Engineering at
the University of Tsukuba Hospital. His technicalinterests include human motion measurement andanalysis and their application for motion andrehabilitation assist devices. Kadone has a PhD in
information science and technology from the
University of Tokyo. Contact him at ______
kadone@ccr.tsukuba.ac.jp.
__________

Hiroko Terasawa is an assistant professor in the Faculty of Library, Information and Media Science at
the University of Tsukuba as well as a JST PRESTO
researcher and the principal investigator of the Laboratory for Sound and People with Computing. Her
research interests include timbre perception, timbrebased data sonification, musical emotion, and interaction in music. Terasawa has a PhD in music from
Stanford University. Contact her at ______
terasawa@
slis.tsukuba.ac.jp.
___________

Kenji Suzuki is an associate professor in the Centerfor Cybernics Research and the principal investigatorof the Artificial Intelligence Laboratory at the University of Tsukuba and a JST PRESTO researcher. His
research interests include assistive and cognitive
robotics, humanoid robotics, augmented humantechnology, biosignal processing, and social playware. Suzuki has a PhD in pure and applied physics
from Waseda University, Japan. Contact him at
kenji@ieee.org.
_________

January–March 2015

masaki@slis.tsukuba.ac.jp.
________________

Masaki Iguchi is an assistant professor at the Tsukuba University of Technology. His research interests
include human motor control, focusing on theapplication of auditory biofeedback for physicallyand visually handicapped people in rehabilitation.
Iguchi has a PhD in physical rehabilitation science_______________________

73
M
q
M
q

M
q

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page qMqM
THE WORLD’S NEWSSTAND®