Sonification of Muscular Activity in Human
Movements Using the Temporal Patterns in EMG
Masaki Matsubara∗ , Hiroko Terasawa† , Hideki Kadone∗ , Kenji Suzuki† and Shoji Makino∗∗ University of Tsukuba, Ibaraki, Japan

E-mail: masaki@tara.tsukuba.ac.jp Tel/Fax: +81-29-8536389† University of Tsukuba, and JST PRESTO

Abstract—Biofeedback is currently considered as an effective
method for medical rehabilitation. It aims to increase the awareness and recognition of the body’s motion by feeding back thephysiological information to the patients in real time. Our goal
is to create an auditory biofeedback that aids understandingof the dynamic motion involving multiple muscular parts, withthe ultimate aim of clinical rehabilitation use. In this paper, we
report the development of a real-time sonification system using
EMG, and we propose three sonification methods that represent
the data in pitch, timbre, and the combination of polyphonictimbre and loudness. Our user evaluation test involves the taskof timing and order identification and a questionnaire about the
subjective comprehensibility and the preferences, leading to a
discussion of the task performance and usability. The results
show that the subjects can understand the order of the muscular
activities at 63.7% accuracy on average. And the sonification
method with polyphonic timbre and loudness provides an 85.2%
accuracy score on average, showing its effectiveness. Regarding
the preference of the sound design, we found that there is not a
direct relationship between the task performance accuracy andthe preference of sound in the proposed implementations.

I. I NTRODUCTION

II. EMG S ONIFICATION
The proposed system configuration is shown in Fig. 1. The
system consists of three steps: EMG measurement, muscular

EMG Measurement→→
Signal Processing→→
Sound Synthesis→

In a medical rehabilitation of motor dysfunction, patients
often have trouble accomplishing the desired motion because
the synchronous (or asynchronous) control of multiple muscular parts in accordance with their intention is difficult. In
order to overcome this difficulty, biofeedback aims to increase
the awareness and recognition of the body motion by feedingback the physiological information to the patients in realtime. In biofeedback, the physiological information is typically
captured using sensors, such as electromyography (EMG),
and visually fed back to the patients [1], [2]. However, someproblems have been pointed out with visual EMG biofeedback:improper posture due to eye distraction, and simultaneouspresentation of multiple-sensor data.
A potential solution to these problems with visual biofeedback is auditory biofeedback, in which the physiologicalinformation is fed back to the users with sound. Auditory
biofeedback typically employs data sonification, which transforms non-audio data into audio information using soundsynthesis. The advantages of auditory biofeedback are (1) there
is no distraction for eyesight, which improves concentration on
the gestural task; (2) audition is more accurate in the senseof timing than vision and thus serves as a better medium
to represent dynamic data; and (3) the cost for the soundsynthesis and audio setup is usually cheaper than that forthe computer graphics with visual displays. Good examples
of EMG sonification have been proposed for elite sports
training [3]; for rehabilitation [4], [5], [6]; and for interactive
music performance [7], [8]. The current issues in sonification,
including the above-mentioned studies, are (1) the evaluation
test by users is often missing; (2) the simultaneous presentationof multiple data streams needs to be investigated, in which
auditory gestalt perception plays the key role; and (3) theuse of multidimensionality of timbre should be exploited in
representing multidimensional data [9].
In our project, the goal is to create an auditory biofeedback
that aids understanding of the dynamic motion involving multiple muscular parts (i.e., muscular activity), with the ultimate
aim of clinical rehabilitation use. In the paper, we report thedevelopment of a real-time sonification system using EMG,
and we propose three sonification methods that represent thedata in pitch, timbre, and the combination of polyphonictimbre and loudness. Our user evaluation test involves the taskof timing and order identification and a questionnaire about the
subjective comprehensibility and the preference, leading to a
discussion of the task performance and usability.

Fig. 1. Proposed sonification system overview.

activity extraction with signal processing techniques, and asound synthesis using a parameter-mapping approach [9]. To
keep the system flexible for a future implementation in realtime, the current approach is in a frame-based processing
design.
A. EMG Measurement
The EMG signals are usually measured using wired or
wireless electrodes. In this study, we employed the wired
measurement technique because of its higher signal-to-noise
ratio, which is well suited to our goal, which is based on EMG
sonification. In the experiments reported in this paper, we used
the g.USBamp biosignal amplifier by g.Tec Inc.(Berryville,
VA). The bus active g.LADYbird electrodes were connectedin bipolar settings on the surface area of the right and left
brachioradialis muscles. The EMG signals were sampled at
1,200 Hz, captured, and stored using the Data Acquisition
toolbox in MATLAB.
B. Signal Processing
The EMG signals contain noise of slow drifts of the directcurrent (DC) component, and they are observed as a series ofpulses. Thus, the raw EMG signals are not suitable for direct
analysis of human movements. In order to extract muscular
activity features, the system removes the DC component, andfilters the signals.
Next, the system calculates features for data sonification.
The purpose of the DC removal is to isolate the signal of amuscle contraction. After the DC removal, resting states aretreated as 0% contractions, and maximum voluntary contractions (MVC) states are treated as 100%. Although the DC
component removal is commonly performed with high-passfiltering, in this study we employed subtraction of an averagepotential in the resting states from the measured EMG signals.
Let 𝑒𝑚 (𝑛) be an EMG signal, 𝑛 the sample, and 𝐸0𝑚 and𝐸𝑚𝑎𝑥𝑚 the mean muscle potentials of the resting and MVC
states at the body part 𝑚, respectively. A muscle potential
amplitude 𝐴𝑚 (𝑛) is calculated by the following equation:
𝐴𝑚 (𝑛) =∣𝑒𝑚 (𝑛) − 𝐸0𝑚 ∣.
𝐸𝑚𝑎𝑥𝑚 − 𝐸0𝑚(1)

Next, the 𝐴˜𝑚 (𝑡) is yielded by smoothing the 𝐴𝑚 (𝑛) with amoving-average filter and upsampling to the same frequency
as the sound output. 𝐴˜𝑚 (𝑡) ranges from 0 to 1, but it usually
falls in the range 0.006 to 0.9 due to the smoothing noise side
effect. The resulting 𝐴˜𝑚 (𝑡) is treated as a feature of muscular
activity in the sound synthesis step described in next sections.
TABLE IM ETHOD FEATURE
MethodA
B
C

Perceived Dynamic Parameter
Pitch
Loudness, Polyphonic Timbre
TimbreSegregation
Yes
Yes
No

C. Sound Synthesis
For the sound synthesis, the acoustic parameters (i.e., amplitude, frequency, and harmonic structure) vary correspondingly
to the muscular activity 𝐴˜𝑚 (𝑡). We implemented three sonification methods—A, B, and C—and Table I shows the featuresof each method.
Method A is a typical technique in sonification studies[9]. It changes the frequency of a basic sound when themuscular activity changes, and the body motions are perceived
as changing in sound pitch. Method B relates the muscular
activity to a combination of loudness and polyphonic timbre,and Method C relates it to timbre transition.
Because of our goal, all methods synthesize the soundobjects in multiple form from each muscle. According to aperspective on the auditory scene [10], sounds are heard asintegrated or segregated auditory objects, even though theyreach the ear as a whole. In methods A and B, the sonified
sounds are heard as segregated into individual components
corresponding to each muscular part. However, in method
C the sonified sounds are heard all together as a balance
between multiple muscular parts. We describe the details ofeach method below.
Method A: This Method relates the muscular activity to an
instantaneous frequency of a sine wave
𝑆(𝑡) = 𝐾𝑀∑sin(2𝜋𝑓𝑚 (𝑡)),

(2)

𝑚=1

where 𝐾 is a coefficient of sound amplitude (𝐾 = 0.45) and
𝑀 is the number of EMG channels. The pitch varies according
to the human movement. When the EMG amplitude becomes
higher, pitch becomes higher, and vice versa. Instantaneous
frequency 𝑓𝑚 (𝑡) is determined by the muscular activity 𝐴˜𝑚 (𝑡),which is calculated by the signal-processing step:𝑓𝑚 (𝑡) = 𝐾𝑚 log2 (1, 000 𝐴˜𝑚 (𝑡)) + 𝑇𝑚 .

(3)

In this study, we set 𝑀 = 2 and use the following constants:
𝐾1 = 50, 𝑇1 = 120, 𝐾2 = 40, and 𝑇2 = 20. These coefficientconstants were adjusted in the preparation phase in order tochange the frequency approximately one octave from 250 Hz
for 𝑚 = 1 and 125 Hz for 𝑚 = 2 (because the muscular
activity 𝐴𝑚 (𝑡) ranges from 0.006 to 0.9 in our configuration).
We used the SuperCollider for the implementation and the Lag
function to change pitch smoothly.
Method B: In this method, we relate the muscular activity to
a combination of loudness and polyphonic timbre. The sounds
are composed by mixing two modulated sounds corresponding
to each muscular part. The muscular activity 𝐴˜𝑚 (𝑡) is regarded
as the weights of the modulated sounds in mixing. The
combination of loudness and polyphonic timbre is graduallychanged, as follows:𝑠(𝑡) =𝑀∑
𝑚=1

𝐴˜𝑚 (𝑡)𝑊𝑚 (𝑡).

(4)𝑊1 (𝑡) =5∑1sin((2𝑖 − 1)𝑓1 𝜋𝑡),
(2𝑖
− 1)
𝑖=1
𝑊2 (𝑡) =

5
∑
1
𝑖=1

2𝑖

sin(2𝑖𝑓2 𝜋𝑡).

(5)
(6)

In order to differentiate the timbre of both two sounds, 𝑊1
is composed of only odd number harmonics corresponding
to the first body part, and 𝑊2 is composed of only evennumber harmonics corresponding to the second body part. In
this study, we set the fundamental frequencies 𝑓1 = 300 Hz
and 𝑓2 = 120 Hz, respectively. The frequency ratio is the sameas 5:2, so that these two sounds form a major third interval.
We implemented this method in MATLAB.
Method C: This method represents a balance of muscular
activity by timbre transition without pitch changing. In orderto realize the timbre transition, the two modulated sounds thathave the same fundamental frequency but different spectral
structures are mixed. This implementation is mostly the sameas method B, but the fundamental frequencies of the both twosounds are set at 𝑓1 = 𝑓2 = 120 Hz. The sonified soundtimbre is gradually changed. When a potential of one bodypart becomes higher, the sonified sound timbre becomes likea clarinet, and when a potential of the other body part becomes
higher, its timbre becomes like the brass instruments.
III. E XPERIMENT
We conducted a listening experiment to verify whether
subjects can understand the muscular activity by listening to
the sonified sounds. As mentioned earlier, the experimental
test involved the task of timing and order identification and aquestionnaire about the subjective comprehensibility and thepreference in case of methods A, B, and C.
A. Experiment Conditions
Participants: Nine subjects (four females and five males)
between 18 and 30 years old (mean 22.7) successfully completed the listening tests. Five of them reported having atleast eight years of experience in performing with musicalinstruments.
Stimuli: We composed acoustic stimuli artificially, which
represent multiple muscular activities. First, we captured an
EMG signal of a motion that involved one muscle contraction.
Second, we duplicated the captured EMG signal with 𝜏
seconds of shifting. These two EMG signals could be regarded
as muscular activity of the 𝛼 and 𝛽 body parts. The EMG
envelopes for stimuli are shown in Fig. 2. Finally, we synthesized the sounds from these envelopes with above-mentioned
sonification method in 𝜏 = {−0.5, −0.2, 0, 0.2, 0.5}. We also
sonified the sounds from the 𝛼 and 𝛽 parts as dummy stimuli.
Environments: All subjects were asked to listen to the soundstimuli via headphones in their usual living spaces.

B. Experiment Procedure
The listening experiment consisted of three parts—
instruction, training, and testing—it lasted approximately 15
minutes. In the instruction phase, all subjects read a detailed
description of the task and the purpose of the experiment. In
the training phase, the subjects listened to sample stimuli, andfamiliarized themselves with the sonification method and thetask, adjusted the volume, and further clarified the test goals
as needed.
In the testing phase, the subjects were asked to identify theorder of the muscular activity and to choose from the following
five statements by listening to the stimuli.
1) Only the 𝛼 part moves.
2) Only the 𝛽 part moves.
3) Both the 𝛼 and 𝛽 parts move synchronously.
4) The 𝛼 part moves before the 𝛽 part moves.
5) The 𝛽 part moves before the 𝛼 part moves.
This identification task used three sets of stimuli corresponding
to the three sonification methods. Each set consisted of 12
stimuli—two times for stimuli with the shifting parameter𝜏 = {−0.5, −0.2, 0, 0.2, 0.5} and once each of two dummy
stimuli, which were sequenced in random order. To avoid theunfairness in each method due to the learning effect of the task,each set was also presented in random order. The subjects wereallowed to listen repeatedly to the stimuli that were hard toanswer, although they were asked to answer intuitively.
After the testing session, the subjects were asked to describe
the process of understanding and their impression about eachmethod, and they were also asked to complete a questionnaire
survey with the following questions about comprehension andpreference:
A. How comprehensible was the sonification sound?
B. What is your preference of the sonification sound?
The evaluation was done with a five-point scale, where 1
corresponded to “bad” and 5 corresponded to “good.”τ = 0.5 (sec.)

Amplitude

The modulated sounds are different in fundamental frequency
and timbre, so they are heard easily as segregated into individual components. These two sounds 𝑊1 (𝑡) and 𝑊2 (𝑡) arecalculated by additive synthesis using the following formula:Time
Fig. 2. EMG envelope for stimuli

αβMethod
A
B
C

Accuracy(%)
62.0(±15.1)
85.2(±11.0)
44.4(±20.5)

Comprehension
3.0(±0.4)
4.3(±0.4)
1.3(±0.4)

Preference
3.9(±1.1)
3.7(±0.8)
1.4(±1.0)

TABLE IIIAVERAGE OF ACCURACY IN MUSICIANS AND NON - MUSICIANS
Accuracy(%)
Non-musician
Musician

Whole
55.6
70.6

A
52.1
70.0

B
77.1
91.7

C
37.5
50.0

τ = −0.5
τ = −0.2
τ = 0.0
τ = 0.2
τ = 0.5

100
Correct Answer Rate (%)

TABLE IIAVERAGE OF ACCURACY, C OMPREHENSION , AND P REFERENCE IN EACH
METHOD (1: BAD , 5: GOOD )

80
60
40
20
0

C. Experiment Results
The experimental results are shown in Tables II and III and
Fig. 3. The average value of accuracy with the confidence interval, comprehensibility (degree of subjective understanding),
and preference in each method are shown in Fig. 2. Table
III shows the mean accuracy of subjects who had musicalexperience and of those who did not have it. Fig. 3 provides
a bar graph of mean accuracy depending on 𝜏 in each method(error bars indicate 95% confidence interval).
IV. D ISCUSSION
A. Comparing the Methods
As shown in Table II, the average of accuracy in method
B marked the highest score at 85.2%, so most subjects could
understand the muscular activity by hearing sounds. As shownin Fig. 3, the accuracies were consistently higher in all casesof 𝜏 in method B than in the others, especially with shortdelay time (±0.2). In particular, as shown in Table III, thesubject who had musical experience identified the timing of
the sounds with different pitches more accurately, so that theaccuracy of method B was 91.7%.
In the interview after the experiment, the subjects wereasked to describe their comprehensibility and preference about
each method. In method A, many subjects felt that their bodywas being moved spontaneously up and down and described
it as “feeling like a rollercoaster ride.” In contrast, in method
B the subjects seemed to interpret the sonified sounds likereading a musical score.
In method C, the subjects had difficulty in distinguishingsounds, whether the sonified sounds were mixed or not. For
that reason, the subjective comprehensibility declined, and thescores for accuracy were low. At this time, the range of timbre
transition was small, so that identifying the order of muscular
activities was difficult. We can increase the comprehensibility
in method C by expanding the difference between the twosounds.
This time, there were two measurement points. However,when the number of body parts increased many pitches would
be mixed, and the sounds might be too complex in methods
A and B. In such cases, method C has a potential advantage
because timbre transition is suitable to represent multidimensional information.

A

B
Sonification Method

C

Fig. 3. The relationship between the accuracy and delay time, in each method

B. Comprehensibility and Preference
As shown in Table II, there is a relationship between accuracy and comprehensibility, but there is no direct relationshipbetween accuracy and the preference of sounds. Among thenine subjects, five preferred method A, three preferred method
B, and one preferred method C. Four of the five subjects who
preferred method A scored their highest accuracies in method
B. Accuracy is considered the highest priority factor forunderstanding the data feature correctly. However, considering
the perspective of usability, it is difficult for long-term use withlow preference even comprehensibility is high. For practical,
long-term use in the future, it is necessary to satisfy both of
the following conditions: sounds must be friendly and clear
for users to understand precisely, and sound familiarity isalso a considerable factor. One of the subjects with musicalbackground said, “I do not prefer any sound, because it is like asound made from a machine, not a musical instrument.” Thesolution to this problem is not only using a familiar timbre
such as an instrument but also examining the suitablenessbetween sonified sounds and data features.
V. C ONCLUSIONSIn this study, we implemented a real-time sonification of
muscular activity, proposed three sonification methods, andconducted the listening experiments with subjects from theperspective of subjective comprehensibility and the preferencein each method. The results show that the subjects canunderstand the order of the muscular activities at 63.7 %accuracy on average, and the sonification with polyphonictimbre and loudness produced an 85.2% accuracy score on
average, demonstrating its effectiveness. Regarding the preference of the sound design, we found that there is not a direct
relationship between the task performance accuracy and thepreference of sound in the proposed implementations. This
causes an interesting future research question, on the trade-off
or balance between preference and efficiency factors.

In our future work, we plan to examine other muscular
activity patterns, such as moving to alternating-pattern andrhythmical-pattern behavior. We also plan to investigate theeffects of increasing the number of body parts, to three or
more. The perception of the sound can be very different in
an interactive setting, in the same way that a simplistic soundeffect in a video game can be highly effective. For that reason,
we would like to move onto an interactive, real-time user
evaluation in the future.
ACKNOWLEDGMENTS
This work was funded by JST PRESTO. We thank Takayuki
Hamano for his help in the implementation of sonification, andTomasz M. Rutkowski for the fruitful discussion. We would
like to thank everyone who kindly participated in our listening
experiment.
R EFERENCES[1] J. S. Petrofsky, “The use of electromyogram biofeedback to reduce
Trendelenburg gait,” European Journal of Applied Physiolgy, vol. 85,
pp. 491–495, 2001.
[2] E. Aiello, D. H. Gates, B. L. Patritti, K. D. Cairns, M. Meister,E. A. Clancy, and P. Bonato. “Visual EMG biofeedback to improve ankle
function in hemiparetic gait,” in Proceedings of Annual International
conference of the IEEE EMBS, pp. 7703–7706, 2005.
[3] N. Schaffert, K. Mattes, and A. O. Effenberg, “A sound design for acoustic
feedback in elite sports,” in Auditory Display, S. Ystad, M. Aramaki,
R. Kronland-Martinet, and K. Jensen, Eds., CMMR/ICAD 2009, Lecture
Notes in Computer Sciences (LNCS) vol. 5954. Berlin:Springer-Verlag,
pp. 143–165, 2010.
[4] L. Chiari, M. Dozza, A. Cappello, F. B. Horak, V. Macellari, and
D. Giansanti, “Audio biofeedback for balance improvements: An
accelerometry-based system,” IEEE Eng Med Biol Mag, vol. 52, no. 12,
pp. 2108–2111, 2005.
[5] M. Dozza, “Biofeedback systems for human postural control,” Ph.D
Thesis, University of Bologna, 2006.
[6] Y. Tsubouchi, and K. Suzuki, “bioTones: A wearable device for EMG
auditory biofeedback,” in Proceedings of Annual International Conferenceof the IEEE EMBS, pp. 6543–6546, 2010.
[7] Y. Nagashima, “Combined force display system of EMG sensor forinteractive performance,” in Proceedings of 2003 International Computer
Music Conference, 2003.
[8] A. Tanaka, “The sound of photographic image,” Artificial Intelligence andSociety, vol. 27, no. 2, pp. 315–318, 2012.
[9] T. Hermann, A. Hunt, and J. G. Neuhoff (Eds.), The Sonification Handbook, Berlin:Logos Verlag, 2011.
[10] A. Bregman, Auditory Scene Analysis: The Perceptual Organization ofSound, Cambridge, MA:MIT Press, 1990.