Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

Interactive Sonification of Gait:
Realtime BioFeedback for People with Parkinson’s Disease
Margaret Schedel1, Daniel Weymouth, Tzvia Pinkhasov, Jay Loomis,
Ilene Berger Morris, Erin Vasudevan, Lisa Muratori2
1

Stony Brook University
Consortium for Digital Arts Culture and Technology
Stony Brook, NY
margaret.schedel@stonybrook.edu

Stony Brook University School of
Medicine
Stony Brook, NY
lisa.muratori@stonybrook.edu

ABSTRACT
Parkinson’s disease (PD) is a progressive neurological illness
characterized by the death of dopaminergic neurons in the basal
ganglia. One of the debilitating aspects of the disease is theinability to generate and maintain internal cues to initiate anddrive movement, particularly in complex tasks such as walking.
Although dopaminergic drug treatments may improve somefeatures of gait, they eventually become ineffective and do notaddress fundamental gait disturbance issues. Therefore,
temporal parameters of gait are left significantly abnormal
despite drug treatment. Sonification represents a novel approachto developing individualized auditory cues based on gait
specific motion analysis data. We propose to use lightweight
sensors that allow three-dimensional tracking of walking withreal-time streaming to a mobile device for individualized
sonification/biofeedback of gait. Initial results testing if people
with PD are capable of recognizing and correcting distortion
cues in pre-recorded music are promising, and are detailed inthis paper.
1.

2

INTRODUCTION TO PD AND MOVEMENT

Parkinson’s disease (PD) is a progressive neurological illness
characterized by the death of dopaminergic neurons in the basal
ganglia. Because the basal ganglia is an important control center
for movement, its degeneration leads to debilitating motor
deficits, with gait disturbances being one of the most common.
The motor symptoms of the disease cannot be eliminated andare typically managed with dopaminergic drugs. Although thesetreatments may improve some features of gait, temporalparameters of walking are left significantly abnormal despite
medication, and asymmetrical walking, as well as episodes offreezing of gait and falling persist [1]. Pharmaceuticals
eventually become ineffective and do not address what
fundamentally underlies these gait disturbances—the inability
to generate and maintain internal cues to initiate and drive
movement.
Since medication has proven to become progressively lesseffective in treating problems with mobility for people with PD,
therapies have developed that use external signals to help
initiate and maintain movement in persons with PD including
cues in the visual, auditory and proprioception domains.
Rhythm, which organizes time into discrete and regular units,
has been shown to be the component of music that mostsignificantly impacts gait in PD. Therapies that use externalcues to help initiate and maintain movement are promising, andmusic therapy in particular seems to address temporal
disturbances effectively [2][3][4]. Imaging studies provide
evidence that sound provokes the simultaneous activation ofboth auditory and motor areas in the brain, and functional MRI
studies conducted in healthy individuals have demonstrated
activation of the basal ganglia in response to isolated metricrhythms, particularly when the rhythm is not strongly suggested
by external cues [5]. Communication between the basal ganglia
and cortical motor areas, such as the premotor andsupplementary motor areas, has also been shown to be more
active while listening to a rhythm [6]. This is suggestive of thebasal ganglia’s role as an internal “clock” and its importance inrhythmic movements such as walking.
Parkinson’s disease patients’ difficulty in self-initiating
movement, demonstrated in various imaging studies [7][8][9],can likely be attributed to the impairment of the basal ganglia asinternal rhythm generators. It has been shown that participants
with PD who were in the early stages of the disease, whenneuronal death tends to occur only within the basal ganglia,
performed significantly worse than control subjects indifferentiating between a regular, rhythmic beat and an irregularbeat [6]. Studies have also shown that movements in response toexternal cues are not impaired in PD [10], and so it may be thatthe impairment in beat perception does not interfere with thebenefits of the therapy.

2.

MUSIC & DISTORTION AS EXTERNAL CUES

Research has shown that sound is more strongly tied totemporal processing than other modalities, and therefore
external auditory cues are more effective than visual cues in
improving gait. This may be due to sound’s stronger ability toenhance cortico-motor connectivity and/or recruit othercompensatory areas such as the cerebellum. It is important tonote that the general auditory deficits see in PD has not beenshow to override the significance of auditory-motor coupling in
improving gait.
Though natural, internal cueing of movement timing is
disturbed by malfunctioning basal ganglia–cortical circuitry inpeople with PD, an external auditory cue in the form of
metronome pulses or rhythmic music can enable affectedindividuals to initiate steps and maintain gait movements[11][12]. In music with a clear beat, the steady temporal input
serves as a continuous reference, creating a rhythmic templatethat influences the motor system’s ability to coordinate andexecute movement [13]. As the pattern of regular external cues
generates temporal expectations, the temporal-motor system
begins to act on those expectations, predicting subsequent beatsand priming movement in anticipation of them [11][13]. In theabsence of a healthy basal ganglia timing system, the94
ISon2016-1

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

cerebellar–thalamic–cortical network seems to be recruited tomediate the entrainment process, or synchronization of
movement to sound (Raglio, 2015; Thaut, 2008; Benoit, Dalla
Bella, Farrugia, Obrig, Mainka and Kotz, 2014; Nombela,2013). Music cueing through the neurologic music therapy
technique known as Rhythmic Auditory Stimulation (RAS) has
been shown to help normalize multiple gait parameters
including velocity, cadence and stride length (Arias & Cudeiro,
2010; McIntosh, 1997; Thaut, McIntosh, Rice, Miller, Rathbun& Brault, 1997).

We measured how long they listened to the undistorted song,
how long it took for them to choose the point where the soundquality matched the original, and the accuracy of the point
where they chose to stop.

Although the temporal aspect of music has been extensively
studied, it is unknown whether more complex cues, such as
combinations of rhythm and pitch distortions correlating to gaiterrors, could be useful for gait improvement. The purpose of
our work is to investigate whether PD patients can perceive
complex auditory cues, which we presented as distortions
superimposed on commercial music, and utilize them formeaningful error correction.
3.

PILOT STUDY – SIMULATED ENVIRONMENT

We experimented with various types of music (Jazz, Bluegrass,
Classical, Pop, Rock, Electronic, Country) and distortions todetermine if certain combinations would result in more effective
external audio cues for gait dysfunction. To maximize
distinction between sounds, we used three types of distortion: 1)
Rhythmic Distortion that creates a jittery, shuddering soundsometimes known as “beat repeat” or “slap-back echo;” 2)
Timbral Distortion that creates a high-frequency whooshing or
warbling sound, using a frequency shifter; and 3) White Noise
which introduces white noise, or static, to mask the sound of themusic during playback. By overlaying these distortions on threecommercial music pieces we created an original auditory
biofeedback method that could be used with gait specificmotion analysis data.
In experiment 1, we examined whether people with PD could
perceive and correct audio distortions on pre-existing music.
Twenty individuals with PD (12 male, aged 52-79 years, x̅ = 67
years) and fifteen healthy peers (7 male, aged 51-89 years, x̅ =66 years) volunteered to participate in this study. All
participants with PD were tested at his or her preferred time of
day to take into account medication ON/OFF fluctuations.
Using an iPad with a custom-designed Lemur interface [15] as
the control surface (see Figure 1) we asked participants to:1) Choose the kind of music they wanted to listen to2) Press play, and adjust the volume if needed
3) Listen to the music undistorted for as long as
they needed to get a sense of the original musicprior to starting an experimental trial.
For each trial subjects manipulated a slide bar whichcorresponded to a randomly generated 120 point distortion
curve with only one ‘zero point’ representing the undistorted
song. Any number away from zero was recorded as error. Eachparticipant performed three trials for each distortion on eachsong to complete 27 trials presented in a random order. Data
was analyzed with a 3 (song) by 3 (distortion) by 2 (group)
repeated measures ANOVA. Significance was set at p<0.05.
Paired t-tests were used to measure multiple comparison effects
using a Bonferroni correction. Variances were assumed to beequal unless the Levene test was violated.

Figure 1. Our iPAD interface designed in the Lemur
App
The Lemur application sent wireless OSC data to Max/MSP[16][17]. Our program in Max recorded the timing information,transformed some of the incoming data, and sent control data to
Ableton Live [18] for playback. All the songs were stored in an
Ableton Live session, and the distortions we used were bundledwith the original software. In addition to the distortions, we hadan extra track of generated white noise that we could fade inover the top of the original signal.
We used the “sound distortion fader” on the iPad to control a
breakpoint function in Max. The slider on the iPad sent valuesof 0-120—this correlated to the X-axis on a function in Max[19]. We randomly created functions with six breakpoints (seeFigure 2). We programmed the function so that only one of thepoints could to be zero on the Y-axis. This point had to between
10 and 90 on the X-axis, and this was the only point that
correlated to zero distortion in the Ableton Live playback. Afterinitial testing we also determined that we needed to ensure thatnone of the other randomly generated numbers were less than
10, making the single point of sonic clarity more obvious. As
the subject moved the slider on the iPad, there were several
areas that had less distortion, but only a single point of complete
clarity.

Figure 2. Randomly Generated Breakpoint Functions
with one zero point on the Y-axis—represented by theopen circle
Figure 3 shows the data we recorded from one subject’s
interaction with the iPad. The X-axis is time in seconds. The Yaxis is amount of distortion (0-120) where zero is no distortionand 120 is maximum distortion. The figure shows that although
the subject located the zero point at 42 seconds, they continued
to experiment with the sound, trying several positions before
returning to the initial zero point and stopping the trial.

95
ISon2016-2

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

200"
0"
35"

45"

55"

65"

Figure 3. Recorded Data collected in Max from iPadinteraction
With this research we were able to show that that patients withneural degeneration from PD can perceive sound distortions andutilize the information for meaningful error correction in asimulated environment with a similar speed and accuracy as
their healthy peers (p>0.05 across conditions). However, we didfind an interaction effect such that individuals with PD mademore errors than control subjects when correcting distortion 1
for songs 1 (p<0.005) and 3 (p<0.001) and distortion 3 for song
3 (p<0.05) (See Figure 4.) Although this suggests that certain
distortions presented more of a challenge to the participants
with PD, it should be noted that the total error was always lessthan 20% of the distortion curve. This means that even the mostinaccurate subject was able to minimize the distortion from apossible 120 point maximum to within 22 points of zero. For
the purposes of biofeedback in gait, this represents a significant
change in behavior. Our next step in a simulated environment isto present subjects with multiple simultaneous distortions todetermine if it is possible to correct for concurrent auditory
effects.

walking. In an effort to provide a personal sonification
biofeedback gait improvement system, we are developing anovel sensor system that will allow three-dimensional tracking
of walking with real-time streaming of gait data to a mobile
device that provides biofeedback through sonification. Thissystem is lightweight so that a person’s gait is not disturbed
while wearing the sensors, and it has an extended battery life sothat there is no need to charge during a typical day. Our systemis engineered to be compatible with commonly used mobile
technology.
We are currently working on feature extraction from the data
sent by the sensors. Eventually we hope to have four variables
which correlate to spatiotemporal aspects of gait including
speed and symmetry. This data can then be used to create
distortions for the user and generate motor learning througherror feedback. Rather than using sound to identify relevant
biological signals, we engage the patient’s perceptual system byintroducing musical error that is intimately linked to deviations
in gait. This biofeedback sonification system is also beneficial
because it allows the patient to have agency in the developmentof their therapy. The individual can choose their music anddevelop their own routine to keep up with their therapy on theirown terms. In addition, several studies show a higher activationof corticomotor areas when patients can choose what music orsounds they want to hear as external audio cues [19]. Therefore,
individualization of music therapy, such the use of patientpreferred music, is more likely to engage the patient and be amore effective therapy. The goal is to create a personalized
training device that leads to long-term improvement in walking
for individuals with PD.

5.

Figure 4. PD vs. Control in Songs 1-3
4.

MOVING FORWARD

Besides its innate temporal nature, music is advantageous formobility therapy because it is pleasurable and can maketreatment less stressful [20], and sound elicits quick reaction
times compared to other sensory cues [21]. This is especially
useful in providing seemingly instantaneous feedback so thatthe patient can adjust and correct their movement, as soon as
they receive the external sound cue that indicates the need forcorrection. Since gait is less automated in PD, it relies moreheavily on cognitive attention, and diverted attention
contributes to greater gait impairment and falls in Parkinson’s
disease patients [21]. Therefore, we are working on a system
which signifies when and how the patient’s gait has been
disturbed, to improve a patient’s gait as well as prevent futurefalls.
Presently there is no commercially available, appropriately
designed tool that can collect gait data, analyze it in real timeand return the data as feedback to the individual while they are

CONCLUSION

In laboratory experiments, highly accurate motion analysis
systems are used for data capture and analysis but these systems
are rarely seen in clinics and they are therefore incompatible
with long-term use by patients in the home or community. Moresimplistic systems, like pressure-sensitive insoles, have beenused to provide step-to-step measures of gait, but these systems
are less accurate and have poor durability. Recently,
laboratories and private companies have turned toaccelerometry or devices equipped with an accelerometer,gyroscope, and/or magnometer to capture larger data sets withgreater accuracy. However, even these systems lack the ability
to stream the data in real-time to a second unit to create abiofeedback system. Furthermore, none of these systems
leverage the power of music to motivate patients to persevere in
their therapy.
Beyond the obvious implications for accessibility, biofeedback
from interactive sonification results in a dynamic anddescriptive portrayal of physiological events, and offers users
the possibility to control and change movement and behavior
the moment that a gait pathology is perceived. PD is adevastating disease that is increasingly common as people live
longer; impaired gait is a particularly troublesome symptom in
PD because it is a predictor of unemployment, increased burdenof care, nursing home placement, falls, morbidity and mortality[23]. Effective treatment of gait impairments can improve
quality of life and decrease healthcare costs. External cueing isan established means of improving walking, but an efficient,
engaging cueing system does not currently exist. We believethat our technology, which uses pre-existing music as a96
ISon2016-3

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

motivator with distortion as an indicator of gait impairment,
instead of systems which create sound from sensors [24], offersa powerful tool in the treatment of movement disorders
associated with PD.
6.
[1]

REFERENCES

N. Giladi, D. McMahon, S. Przedborski, E. Flaster, S.
Guillory, V. Kostic and S. Fahn, "Motor blocks in
Parkinson's disease", Neurology, vol. 42, no. 2, pp. 333333, 1992.
[2] M. de Dreu, A. van der Wilk, E. Poppe, G. Kwakkel andE. van Wegen, "Rehabilitation, exercise therapy andmusic in patients with Parkinson's disease: a metaanalysis of the effects of music-based movement therapy
on walking ability, balance and quality oflife", Parkinsonism & Related Disorders, vol. 18, pp.
S114-S119, 2012.
[3] G. McIntosh, S. Brown, R. Rice and M. Thaut,"Rhythmic auditory-motor facilitation of gait patterns inpatients with Parkinson's disease.", Journal of Neurology,
Neurosurgery & Psychiatry, vol. 62, no. 1, pp. 22-26,
1997.
[4] C. Nombela, L. Hughes, A. Owen and J. Grahn, "Into thegroove: Can rhythm influence Parkinson's
disease?", Neuroscience & Biobehavioral Reviews, vol.
37, no. 10, pp. 2564-2570, 2013.
[5] B. Haslinger, P. Erhard, E. Altenmüller, U. Schroeder, H.
Boecker and A. Ceballos-Baumann, "Transmodal
Sensorimotor Networks during Action Observation in
Professional Pianists", Journal of Cognitive
Neuroscience, vol. 17, no. 2, pp. 282-293, 2005.
[6] J. Grahn, "The Role of the Basal Ganglia in Beat
Perception", Annals of the New York Academy of
Sciences, vol. 1169, no. 1, pp. 35-45, 2009.
[7] T. Wu, L. Wang, M. Hallett, Y. Chen, K. Li and P. Chan,"Effective connectivity of brain networks during selfinitiated movement in Parkinson's disease", NeuroImage,
vol. 55, no. 1, pp. 204-215, 2011.
[8] E. Playford, I. Jenkins, R. Passingham, J. Nutt, R.
Frackowiak and D. Brooks, "Impaired mesial frontal andputamen activation in Parkinson's disease: A positronemission tomography study",Annals of Neurology, vol.
32, no. 2, pp. 151-161, 1992.
[9] T. Wu, P. Chan and M. Hallett, "Effective connectivity ofneural networks in automatic movements in Parkinson's
disease", NeuroImage, vol. 49, no. 3, pp. 2581-2587,
2010.
[10] K. bötzel and s. schulze, "Self-initiated versus externallytriggered movements. I. An investigation using
measurement of regional cerebral blood flow with PET
and movement-related potentials in normal andParkinson's disease subjects", Brain, vol. 119, no. 3, pp.
1045-1046, 1996.
[11] C. Benoit, S. Dalla Bella, N. Farrugia, H. Obrig, S. Mainka
and S. Kotz, "Musically Cued Gait-Training Improves
Both Perceptual and Motor Timing in Parkinsonâ€™s
Disease", Frontiers in Human Neuroscience, vol. 8, 2014.
[12] S. Mainka, "Music stimulates muscles, mind, and feelingsin one go", Frontiers in Psychology, vol. 6, 2015.
[13] C. Nombela, L. Hughes, A. Owen and J. Grahn, "Into thegroove: Can rhythm influence Parkinson's disease?",Neuroscience & Biobehavioral Reviews, vol. 37, no. 10,
pp. 2564-2570, 2013.

[14] M. Rodger, W. Young and C. Craig, "Synthesis ofWalking Sounds for Alleviating Gait Disturbances in
Parkinson's Disease", IEEE Trans. Neural Syst. Rehabil.
Eng., vol. 22, no. 3, pp. 543-548, 2014.
[15] 2016. [Online]. Available:
https://liine.net/en/products/lemur/. [Accessed: 23- Sep2016].
[16] M. WRIGHT, "Open Sound Control: an enabling
technology for musical networking", Organised Sound,vol. 10, no. 03, p. 193, 2005.
[17] Cycling74.com, 2016. [Online]. Available:
https://cycling74.com/products/max/. [Accessed: 23- Sep2016].
[18] "Learn more about our music making software Live |Ableton", Ableton.com, 2016. [Online]. Available:
https://www.ableton.com/en/live/. [Accessed: 23- Sep2016].
[19] "Max 7 - function Reference", Docs.cycling74.com, 2016.
[Online]. Available:
https://docs.cycling74.com/max7/maxobject/function.
[Accessed: 23- Sep- 2016].
[20] A. Blood and R. Zatorre, "Intensely pleasurable responses
to music correlate with activity in brain regions
implicated in reward and emotion", Proceedings of the
National Academy of Sciences, vol. 98, no. 20, pp. 1181811823, 2001.
[21] M. Thaut, G. Kenyon, M. Schauer and G. McIntosh, "Theconnection between rhythmicity and brain
function", IEEE Engineering in Medicine and Biology
Magazine, vol. 18, no. 2, pp. 101-108, 1999.
[22] G. Yogev, N. Giladi, C. Peretz, S. Springer, E. Simon and
J. Hausdorff, "Dual tasking, gait rhythmicity, andParkinson's disease: Which aspects of gait are attention
demanding?", European Journal of Neuroscience, vol.
22, no. 5, pp. 1248-1256, 2005.
[23] D. Muslimovic, B. Post, J. Speelman, B. Schmand and R.
de Haan, "Determinants of disability and quality of life inmild to moderate Parkinson disease", Neurology, vol. 70,
no. 23, pp. 2241-2247, 2008.
[24] B. Horsak, R. Dlapka, M. Iber, A. Gorgas, A. Kiselka, C.
Gradl, T. Siragy and J. Doppler, "SONIGait: a wireless
instrumented insole device for real-time sonification ofgait", Journal on Multimodal User Interfaces, vol. 10, no.
3, pp. 195-206, 2016.

97
ISon2016-4