# PDPuzzleTable: A Leap Motion Exergame for Dual-Tasking Rehabilitation in Parkinson's Disease. Design and Study Protocol

Augusto Garcia-Agundez, Mareike Goosses, Robert Konrad, Manuel Stork, Hagen Becker, Stefan Göbel, Elke Kalbe

► To cite this version:

Augusto Garcia-Agundez, Mareike Goosses, Robert Konrad, Manuel Stork, Hagen Becker, et al. PDPuzzleTable: A Leap Motion Exergame for Dual-Tasking Rehabilitation in Parkinson's Disease. Design and Study Protocol. 1st Joint International Conference on Entertainment Computing and Serious Games (ICEC-JCSG), Nov 2019, Arequipa, Peru. pp.402-406, 10.1007/978-3-030-34644-7_35. hal-03652033

HAL Id: hal-03652033

https://inria.hal.science/hal-03652033

Submitted on 26 Apr 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# PDPuzzleTable: A Leap Motion Exergame for Dual-Tasking Rehabilitation in Parkinson's Disease. Design and Study Protocol

Augusto Garcia-Agundez<fnref n="1" />, Mareike Goosses<fnref n="2" />, Robert Konrad<fnref n="1" />, Manuel Stork<fnref n="1" />, Hagen Becker<fnref n="1" />, Stefan Göbel<fnref n="1" /> and Elke Kalbe<fnref n="2" />

Augusto.garcia@kom.tu-darmstadt.de

**Abstract.** In this paper, we present PDPuzzleTable, our exergame framework for dual-tasking rehabilitation exercises for patients with Parkinson's Disease (PD). It is our aim to create a home monitorization scenario for the Leap Motion Sensor together with a specific set of exercises designed for PD, that allows us to follow upon disease progression by inferring motor-cognitive skills remotely and passively.

**Keywords:** Parkinson's Disease, Cognitive Impairment, Motor Impairment, Rehabilitation, Exergames, Serious Games, Leap Motion

## 1 Introduction

Parkinson's Disease (PD) is a neurodegenerative disease primarily caused by the idiopathic degeneration of the dopaminergic neurons in the substantia nigra pars compacta. The core symptoms of PD are bradykinesia, tremor, rigor and postural instability. Besides motor symptoms, cognitive symptoms are also typical of PD [1], particularly executive dysfunction [2]. Once objectified, it is called Mild Cognitive Impairment in PD (PD-MCI) and presents itself as a subtle difficulty on complex functional tasks. The prevalence of PD-MCI is around 25% [3] and it is a risk factor for decline into PD Dementia (PDD). PDD excludes PD patients from certain treatments such as deep brain stimulation, and currently has no approved pharmacological intervention.

However, recent research shows cognitive training may improve or stabilize the cognitive skills of affected patients [4]. This research also indicates that transfer effects can be expected, that is, cognitive training may improve motor symptoms and vice versa [5]. An excellent way to combine cognitive training with motor rehabilitation is the use of exergames, i.e. games that require the player to perform physical movements. Another advantage of this approach is the sensory feedback provided by game controllers and sensors [6]. A recent systematic review [7] indicated the significant

<footnote n="1">¹Multimedia Communications Lab (KOM) TU Darmstadt. Darmstadt, Germany</footnote>

<footnote n="2">²Medical Psychology | Neuropsychology and Gender Studies & Center for Neuropsychological Diagnostics and Intervention (CeNDI). Faculty of Medicine and University Hospital Cologne Cologne, Germany</footnote>
progress of exergames for PD patients in the recent years, also pointing to the requirements of future work, mainly the standardization of outcome evaluation and the inclusion of new sensors and control techniques to train unaddressed areas, such as fine motor skills.

For this purpose, the Leap Motion Sensor (LMS) seems to be a highly promising approach, since it provides an intuitive, natural interface while providing sensory feedback on motor skills. The LMS has already been shown to have potential in evaluating the motor performance of PD patients [8, 9], particularly for hand opening and closing exercises or finger tapping [10]. In fact, researchers have already begun developing LMS based exergames or LMS-digitalized version of traditional motor skill assessment methods such as the Fugl-Meyer test or the Box and Blocks test [11].

## 2 Methods

As a first step, we considered the diverse cognitive symptoms of PD and how they can be trained via LMS Exercises. Table 1 summarizes our observations in this regard.

<table><caption>Table 1: Cognitive Symptoms of PD and potential training in LMS scenarios</caption><thead><tr><td>PD Symptom</td><td>Treatment Intent</td></tr></thead><tbody><tr><td>Resting Tremor</td><td>Fine motor training in ADL-Context</td></tr><tr><td>Reduction of Joint Mobility</td><td>Grab-and-drop exercises, writing</td></tr><tr><td>Coordination/Speed issues</td><td>Training in a virtual scenario</td></tr><tr><td>Executive Function</td><td>Calculation and manipulation exercises with increasing difficulty</td></tr><tr><td>Concentration problems</td><td>Search and differentiate, sorting, simplification exercises</td></tr><tr><td>Memory disturbances</td><td>Memorization techniques (systematic repetition, method of Loci). Memory Games/Word Puzzles</td></tr><tr><td>Delay in cognitive processes</td><td>Awareness exercises, Speed exercises</td></tr></tbody></table>

In order to create our exergaming framework, we accessed the LMS data directly by using LeapC, a C-style Application Programming Interface (API) for the LMS. This API was connected to our application, developed under Kha, a Haxe-based open source multimedia framework. By using LeapC, it is possible to access the raw data of the LMS directly, as presented in Table 2. Besides palm and finger coordinates, the LMS is capable of measuring “pinch” and “grab” parameters, which determine the degree of closure of a pinching motion (that is, grabbing an object with the index finger and thumb) and grabbing motion (closing the fist) respectively. This provides a great advantage when developing scenarios for people with varied fine motor mobili-
ty, since the pinch and grab parameter thresholds can then be adjusted to adapt to
users with limited mobility, as it occurs in PD.

The main advantage of the LMS, besides acquiring hand position with a sampling rate of 200 Hz, is precisely this grabbing and pinching motion detection, since it allows developers to adapt their environments to the varied motor skills of their target group [11]. For this purpose, we implemented to unitary parameters, one for pinching and one for grabbing, 1 being a completely closed pinch or grab. This permits us to demand specific movements for patients with determinate degrees of hand closure, which we can quickly change to ease playability or increase difficulty with time to further improve the motor skills of players.

Considering our observations, we decided to create two scenarios. Firstly, the Tower of Hanoi (ToH), also called tower of London (Figure 1). The goal in this game is to move a set of n discs from the left to either the middle or right columns in the shortest possible time. It is only possible to place a disc over an empty tower or a larger disc, one by one, which limits the possible movements. The game can be adapted to operate with pinching motions, grabbing motions, or both. This task combines motor skills with problem solving and sequencing, two important cognition areas.

<figure><img src="image_2.png" /><figcaption>Fig. 1: Example images of the TOH game</figcaption></figure>

Secondly, we decided to include a Simon-based Memory Game (SMG), in which the player has to follow a visual and musical sequence by clicking on blocks. This allows the training of working memory and sequencing. For the SMG, the goal is to grab or pinch the blocks, following a randomized music and color sequence generated by the game, that is one element longer for every successfully completed sequence. The sequence can be limited to music or color exclusively and it may be requested from the player to repeat the sequence forwards or backwards. This interaction pattern was chosen purposely to mimic a hand opening/closing test. Figure 2 presents the SMG game.

<figure><img src="image_3.png" /><figcaption>Fig. 2: Example Image of the SMG game</figcaption></figure>
<table><caption>Table 2: LMS and Game extracted Parameters</caption><thead><tr><td>Hand Parameters (200 Hz)</td><td>Game Parameters (Per playthrough)</td></tr></thead><tbody><tr><td>Fingertip position (x,y,z) of each finger</td><td>Number of movements / Max Sequence Length</td></tr><tr><td>Hand palm position (x,y,z)</td><td>Elapsed Time (s)</td></tr><tr><td>Hand palm rotation (x,y,z)</td><td>Input Timelapse (s)</td></tr><tr><td>Pinch ratio (0-1)</td><td>Number of disks (TOH) / Number of Elements (SMG)</td></tr><tr><td>Grab ratio (0-1)</td><td></td></tr></tbody></table>

For every game session, relevant game data as well as finger data are saved for future processing. This data is described in Table 2, and is saved both locally (as a .csv file) and remotely using a Rest API if desired. The data can be then automatically processed in Matlab and relevant parameters can be extracted, for example detected tremor (if any), variations in elapsed time or number of movements across different sessions, or maximum fist closing / pinching achieved across different sessions. On a future pilot test, we plan to link these game and sensor parameters to relevant PD assessment scores, such as UPDRS-III (motor examination) to create difficulty adaptation algorithms as well as a potential home monitoring scenario [12]. To evaluate the scenario, we firstly plan to analyze its feasibility, considering its technical, therapeutic and patient-centered outcomes. Followed by this is a cross-over evaluation, meaning the PD patient sample will be divided into two groups: Group 1 will first test a physically demanding-but cognitively lenient version and then a cognitively demanding but physically easy one, while group 2 will test the same versions in the opposite order. A diagram of the study protocol is presented on Figure 2. The goal of this phase is to determine whether the order in which the cognitive or motor task is presented plays a role.

<figure><img src="image_4.png" /><figcaption>Fig. 3: Study Protocol</figcaption></figure>

## References

[1] D. Muslimović, B. Post, J. D. Speelman, and B. Schmand, "Cognitive profile of patients with newly diagnosed Parkinson disease," *Neurology*, vol. 65, no. 8, pp. 1239-1245, 2005.
[2] E. Kalbe et al., "Subtypes of mild cognitive impairment in patients with Parkinson's disease: evidence from the LANDSCAPE study," pp. jnnp-2016-313838, 2016.

[3] D. Aarsland, K. Brønnick, and T. Fladby, "Mild cognitive impairment in Parkinson's disease," *Current neurology and neuroscience reports*, vol. 11, no. 4, pp. 371-378, 2011.

[4] I. H. Leung, C. C. Walton, H. Hallock, S. J. Lewis, M. Valenzuela, and A. J. N. Lampit, "Cognitive training in Parkinson disease: A systematic review and meta-analysis," p. 10.1212/WNL. 000000000002145, 2015.

[5] C. C. Walton, J. M. Shine, L. Mowszowski, S. L. Naismith, and S. J. Lewis, "Freezing of gait in Parkinson's disease: current treatments and the potential role for cognitive training," *Restorative neurology and neuroscience*, vol. 32, no. 3, pp. 411-422, 2014.

[6] A. Garcia-Agundez, A.-K. Folkerts, R. Konrad, P. Caseman, S. Göbel, and E. Kalbe, "PDDanceCity: An Exergame for Patients with Idiopathic Parkinson's Disease and Cognitive Impairment," *Mensch und Computer 2017-Tagungsband*, 2017.

[7] A. Garcia-Agundez et al., "Recent advances in rehabilitation for Parkinson's Disease with Exergames: A Systematic Review," vol. 16, no. 1, p. 17, 2019.

[8] A. H. Butt, E. Rovini, C. Dolciotti, P. Bongioanni, G. De Petris, and F. Cavallo, "Leap motion evaluation for assessment of upper limb motor skills in Parkinson's disease," in *2017 International Conference on Rehabilitation Robotics (ICORR)*, 2017, pp. 116-121: IEEE.

[9] H. Kaji and M. Sugano, "A noncontact tremor measurement system using leap motion," in *Proceedings of the 6th International Conference on Informatics, Environment, Energy and Applications*, 2017, pp. 76-79: ACM.

[10] A. Butt et al., "Objective and automatic classification of Parkinson disease with Leap Motion controller," vol. 17, no. 1, p. 168, 2018.

[11] E. D. Oña, C. Balaguer, R. Cano-de la Cuerda, S. Collado-Vázquez, and A. Jardón, "Effectiveness of Serious Games for Leap Motion on the Functionality of the Upper Limb in Parkinson's Disease: A Feasibility Study," *Computational intelligence and neuroscience*, vol. 2018, 2018.

[12] A. Garcia-Agundez, S. Sharma, T. Dutz, and S. J. M. u. C. W. Göbel, "Ein Smartphone-basiertes Framework für Patientenfernüberwachung," 2016.