# Monitoring User's Brain Activity for a Virtual Coach

Bram van de Laar, Anton Nijholt, Job Zwiers

► To cite this version:

Bram van de Laar, Anton Nijholt, Job Zwiers. Monitoring User's Brain Activity for a Virtual Coach. 9th International Conference on Entertainment Computing (ICEC), Sep 2010, Seoul, South Korea. pp.511-513, 10.1007/978-3-642-15399-0_76. hal-01059022

HAL Id: hal-01059022

https://inria.hal.science/hal-01059022

Submitted on 29 Aug 2014

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Monitoring User's Brain Activity for a Virtual Coach

Bram van de Laar¹, Anton Nijholt¹, Job Zwiers¹

¹ Human Media Interaction, University of Twente, Enschede, The Netherlands
{laar, nijholt, zwiers}@ewi.utwente.nl

**Abstract.** The system described in this paper is an attempt at developing a coach for sports using a virtual world and multimodal interaction, including brain activity. Users can ride a bicycle through a virtual world while the coach monitors the user's performance. The system incorporates the user's brain activity, heart rate and respiration rate. These data are analyzed and the features are sent through to give the virtual coach the instructions for movements and dialogues to coach the user. The Electroencephalogram (EEG) provides ample possibilities to research the brain activity of the user and to provide for an extra modality in the interaction.

**Keywords:** BCI, Coaching, 3D Avatar, Multimodal Interaction, Sports, Extreme EEG

## 1 Introduction

Finding motivation for the practice of sports and bodily exercises is becoming harder and harder in the fast-paced society we live in. A lot of people would therefore be helped by the aid of a virtual coach [3]. Earlier work by Ruttkay et al. [7] provides the basis for virtual coaches. The system proposed in this paper is a start for such a virtual coach using brain activity as a modality. In the first working version of this system we focus on a bicyclist, exercising on a real bicycle that has been mounted onto a VR bike frame.

In this system we intend to use brain activity, various physiological signals such as heart rate, respiration rate and volume as well as pedal rate and power exerted by the user on the bicycle.

While heart and respiration rate are quite common features to measure on what level the user is performing, brain activity is not. However, research by Palva et al. [6] and Schürmann et al. [8] indicates that interesting meaning can be found in the alpha band of the electroencephalogram (EEG). Energy in the alpha band can indicate how conscious and attentive the subject is to a task. Also the work by Klimesch et al. [5] provides interesting cues for more research into different energy bands in the EEG of an exercising user. In future research related to this paper we will investigate the use of such information for a better informed virtual coach.
## 2 Overview of the system

The system as a whole is built up from different components as depicted in figure 1. These components constitute a loop in which sensory data from the user is measured, analysed and behaviour and dialogues for the virtual coach selected. After this the avatar component realizes the behaviour and dialogue which the user can see on a screen and hear through the loudspeakers.

The first step in the loop is to take the EEG and physiological data. These are acquired from various sensors, located on the user's body. The data is then streamed in real-time to the signal analysis component. This component interprets the data and sends the features to the behaviour selection component. Depending on the value of the incoming features this component selects the most fitting behaviour for the coach. Then in turn, the dialogue system selects the dialogue to send to the Virtual coach. The virtual coach carries out the dialogue and movements which are the feedback for the user.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Overview of the Virtual Coach system</figcaption></figure>

The user can ride through any virtual world which supports the VR bicycle frame. Users sitting on this bike can then use the steer and pedal to navigate through this world, while being monitored and coached by the system. The avatar used for the virtual coach consists of the Elckerlyc system as introduced by Van Welbergen et al. [9]. The Behaviour selection and Dialogue component communicate through Behaviour Markup Language (BML). Elckerlyc visualizes the movements and dialogues on the screen.

## 3 Analysis and use of neurophysiological data

In the current version of the system the EEG is analysed for energy in the alpha band on a certain part of the brain, namely location 'Pz' in the 10-20 placement system [2]. The feature measured on this location on the scalp is an indication for the user either being agitated and alert or relaxed and distracted. The EEG is band pass filtered between 8 and 13 Hz after which the band power is calculated. A moving window of 120 seconds is used to normalize the output, id est, the extrema and the mean of the signal over the last 120 seconds determine the actual value of this feature. This way, the algorithm will adapt to any user using the system, invariable of the baseline power in the alpha band of a user.
The interpretation of the user's mental state is then used to select the appropriate attitude for the virtual coach towards the user. The heart and respiration rate will be used to monitor if the user is performing on the right level, depending on the goal being weight loss or condition improvement.

Because the user is performing physical activity, robustness of the EEG signal quality and classification of said signal is very important. However, similar research by Katsis et al [4] in race car drivers indicates that this should be possible.

## 4 Conclusion and Future Work

In this paper a system for a virtual coach using brain activity is introduced. Research into how brain activity can be put to use for a virtual coach is one of the main interests.

While currently the system only uses one feature in the EEG, future research will include using more complex or multiple features to provide the virtual coach with a more comprehensive representation of the user's mental and physical state. Such features could focus on measuring the flow of the user as defined by Cziksentmihalyi [1]. Ultimately this could lead to a system that keeps track of the user being in the flow and adapts itself to provide a better user experience.

## References

1. Cziksentmihalyi, M. Flow: The Psychology of Optimal Experience. Harper & Row Publishers (1990)

2. Homan, R. W., Herman, J., & Purdy, P. Cerebral location of international 10-20 system electrode placement. Electroencephalography and clin. neurophys., 66(4), 376—82 (1987)

3. IJsselsteijn, W.A., de Kort, Y.A.W., Bonants, R., de Jager, M., Westerink, J. Virtual Cycling: Effects of immersion and a virtual coach on motivation and presence in a home fitness application. Proceedings Virtual Reality Design and Evaluation Workshop 2004, University of Nottingham, UK (2004)

4. Katsis, C., Katertsidis, N., Ganiatsas, G., & Fotiadis, D. Toward Emotion Recognition in Car-Racing Drivers: A Biosignal Processing Approach. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 38(3), 502—512 (2008)

5. Klimesch, W., Doppelmayr, M., Russegger, H., Pachinger, T., & Schwaiger, J. Induced alpha band power changes in the human EEG and attention. Neuroscience Letters, 244(2), 73—76 (1998)

6. Palva, S., & Palva, J. M. New vistas for alpha-frequency band oscillations. Trends in neurosciences, 30(4), pp. 150—158 (2007)

7. Ruttkay, Z., Zwiers, J., Welbergen, H. V., & Reidsma, D. Towards a Reactive Virtual Trainer. In J. Gratch et al. (Eds.): Intelligent Virtual Agents, 292—303. Springer, Heidelberg (2006)

8. Schürmann, M., & Basar, E. Update article: Alpha oscillations shed new light on relation between EEG and single neurons. Neuroscience Research, 33, 79—80 (1999)

9. Van Welbergen, H., Reidsma, D., & Zwiers, J. A Demonstration of Continuous Interaction with Elckerlyc. In 3rd Workshop on Multimodal Output Generation (2010)