# Tailored, Multimodal and Opportune Interactions on a Wearable Sport Coach: The WE-nner Framework

Jean-Claude Martin, Céline Clavel

<figure><img src="image_1.png" /></figure>

## To cite this version:

Jean-Claude Martin, Céline Clavel. Tailored, Multimodal and Opportune Interactions on a Wearable Sport Coach: The WE-nner Framework. 16th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2017, Bombay, India. pp.24-32, 10.1007/978-3-319-92081-8_3. hal-01821423

HAL Id: hal-01821423

https://inria.hal.science/hal-01821423

Submitted on 22 Jun 2018

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Tailored, Multimodal and Opportune Interactions on a Wearable Sport Coach: the WE-nner Framework

Jean-Claude Martin and Céline Clavel

LIMSI-CNRS, Université Paris-Sud, Orsay, France
MARTIN@LIMSI.FR

**Abstract.** A growing body of evidence from Psychology and Sport Sciences shows that physical activity can be a cost-effective and safe intervention for the prevention and treatment of a wide range of mental and physical health problems. Research in domains such as the Internet of Things (IoT), wearables and persuasive technologies suggest that a coach intended to promote physical activities needs to provide personalized interaction. In this paper we introduce the WE-nner (pronounce “winner”) framework for designing an automated coach promoting physical activity which supports interactions between the user and a smart wearable that are: 1) personalized to the user, 2) dynamic (e.g. occurring during a physical activity), and 3) multimodal (e.g. combine graphics, text, audio and touch). We explain how we implemented this framework on a commercial smart-watch and provide illustrative examples on how it provides dynamic personalized and multimodal interactions considering features from user’s profile. Future directions are discussed in terms of how this framework can be used and adapted to consider theories and models from Psychology and Sport Sciences.

**Keywords:** Wearables, Embedded Computing, Physical Activity, Personalization, Multimodal Interaction, Sport Coach.

## 1 Introduction

Researchers are designing *automated exercise coaches* for health [6, 7] and sport experts or beginners [10]. A growing body of evidence from Psychology and Sport Sciences shows that physical activity can be a cost-effective and safe intervention for the prevention and treatment of a wide range of mental and physical health problems [1]. Yet, the design of virtual health agents faces multiple challenges such as [9]: (1) interpreting the situation and people's intentions, (2) intervention reasoning, (3) generating informative, educative, persuasive computer behavior, and (4) engineering generic solutions. Research in domains such as the Internet of Things (IoT), wearables and persuasive technologies [2] does suggest that a coach intended to promote physical activities needs to provide *tailored interaction* [8, 17], *personalized messages* and bring into play persuasive strategies that depend on gender and personality [3], and stage of be-
behavior change [4]. The use of persuasive technologies and virtual coaches for promoting physical activity might help stress management [5], but adhesion might also depend on aspects of the self [15, 16].

Thus *tailoring information* is significantly more effective when it provides *specific* details on how to improve the performance, rather than when it just indicates whether the performance is correct or not [30]. When tailoring information involves emotional dimension, interaction becomes more attractive, increasing student engagement and motivation [29, 31]. Finally, when users manage the tailoring process, Kim et al [32] observe in a positive attitude toward the conveyed information.

Although lots of people still use a phone for collecting data during a physical activity, *smartwatches* display several advantages over mobile phones: they are closer to the body, safer to use than the mobile phone when moving, and they are designed to have a longer battery life. Using smartwatches to support affective interaction nevertheless rises challenges in terms of human-computer interaction. Multiple commercial wearables for sport activities are available. We claim that in order to become smart, an automated coach requires to provide via a smartwatch: 1) interactions that are personalized to the user, 2) dynamic support occurring not only before and after a physical activity but also during a physical activity, and 3) multimodal interactions (including subtle and complementary use of embedded graphics, audio and touch).

In this paper, we introduce WE-nner, a framework for the design of a coach supporting personalized interaction for physical activities. We illustrate how we implemented this coach using a commercial platform for embedded programming. We discuss how this framework impacts the relation between users and computing devices (smartwatch, mobile phone, web site) and may improve the relationships between users and computing devices.

## 2 WE-nner framework

Brinkman [9] proposes a research framework for behavior change support systems that brings into play situation interpretation, intervention reasoning (using personal and population data; possibly calling for remote assistance from human health professionals), and the generation of informative, educative and persuasive behaviors. In our framework we focus on this last step which requires that the system handles relevant features of user's profile and is able to use it dynamically during a physical activity to generate personalized and multimodal interactions.

### 2.1 Multimodal interaction during physical activity

The WWHT framework [19] describes how to present multimodal information to users along several questions: What is the information to present? Which modalities should we use to present this information? How to present the information using these modalities? How to handle the evolution of the resulting presentation?

Few studies about multimodal interaction were conducted with smartwatches during physical activity. Lee et al. designed multimodal and mobile interaction between a user and a sport-watch while the user is walking on a treadmill [14].
Interaction with a sport-watch can be personalized along several dimensions:

*   **Time**: interaction can take place with the user at different timescales around a physical activity, for example several days before an activity, just before the activity starts, during the activity, just after the end of the activity, and several hours or days after the activity. One might expect that interactions at these different times would have different goals and results

*   **Modality**: information can be presented to the user on several channels and modalities: on the visual channel (graphics, text messages, text menus: displayed on the screen), touch (vibrations), and audio signals

*   **Content**: technical messages, motivating message, warning messages,

*   **Goal**: the task of coaching a user involves several components and use cases (e.g. weekly burnt calories)

Currently, user's profile include the user's name, gender, history of activities and weekly goal in terms of burnt calories.

## 2.2 Software implementation

Few smartwatches provide a software Development Kit and a programming language that enables embedded computing on the smartwatch. This is nevertheless required to dynamically customize the interaction with users. This allows to design customized interactions that fit between minimal interactions available on other commercial wearables and full symbiosis that are investigated in research.

We implemented the WE-nner framework using the GARMIN Connect IQ environment and its MonkeyC object oriented programming language. GARMIN is one of the few smartwatch manufacturer that provides a Software Development Kit that enables to develop a program on a PC and to upload and embed the software on the watch itself. GARMIN offers a dedicated programming environment called Connect IQ<fnref n="1" /> which enables to program and upload several kinds of applications on a smartwatch (face, widget, data field and apps). Different types of applications have different degrees of access to the sensors and actuators of the watch and allow for different types of interactive capabilities. GARMIN wearables support four types of interactions components:

*   **WatchFaces** provide personalized “passive” displays of the main screen. Users are able to choose and download the watch face they prefer and add any information they like.

*   **Widgets** provide at-a-glance information to the user that meets the individual customization. They are usually small practical tools like a compass or a weather report and are limited in terms of interaction capabilities.

*   **DataFields** are fields displaying data which is computed or available at runtime (e.g. speed, time). The expert user can select the fields that she wants to be displayed and their order of presentation. DataFields do not support any interaction.

<footnote n="1">¹ https://developer.garmin.com/connect-iq/</footnote>
* Apps are the most interactive components that can be uploaded on the watch. They can contain menus, data, textual and graphical messages that can be selected and combined at runtime. An app is explicitly started from the main menu of the watch.

Interaction with advanced sport smartwatches such as a GARMIN watch can be quite tricky for the ordinary user. Furthermore, even apps available on the GARMIN store do not provide flexible and personalized multimodal interactions.

We implemented our WE-nner framework as a Connect IQ App in order to benefit from the maximum access to sensors and actuators. Fig. 1 illustrates how the WE-nner software uploaded on the GARMIN Fenix 5 smartwatch provides interaction that is personalized at runtime before an activity starts. The smartwatch collects information from an .xml file detailing user's profile (e.g. user name, user birth date, activity history). The menus as well as the responses of the different physical buttons that surround the watch can be completely changed at runtime to cope with a given user.

<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Screendumps showing personalized interaction before an activity starts: a) WE-nner displays a personalized message including user's name, b) it suggests several activities (which can consider user's history of previous activities), c) the user is able to select the activity she wants to do, and d) the activity is ready to start.</figcaption></figure>

Interaction can also be personalized *during* a physical activity. For example, a warning text, audio and tactile message (vibrations) can be generated dynamically by WE-nner when the heart rate frequency goes beyond a given percentage of the current user's maximum heart rate frequency (Fig. 2).

Finally, WE-nner also enables to personalize interaction after the end of the activity, and display for example a congratulation message that embeds user's name and information about the current achievements in terms of burnt calories (Fig. 3).

Fig. 4 illustrates the displays on the watch during an outdoor test.
<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Screendumps showing personalized interaction *during* an activity: a) at some point during an activity, a multimodal (graphics, audio, vibrations) personalized message is dynamically computed and displayed including user's name and considering a threshold of heart rate frequency that is specific to this user.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Screendumps showing personalized interaction *after* an activity: a) a personalized message is computed at runtime on the watch and includes user's name and the number of burnt calories during the activity, and b) the percentage of this user's weekly calories goals can be graphically displayed.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>**Fig. 4.** Personalized messages displayed on the smartwatch during an outdoor test.</figcaption></figure>
## 2.3 Software architecture

The smartwatch provides sensors, screen and loudspeaker that enables a user to track her daily physical activities. The activity information that includes time, duration, speed, distance, calories burned and heart rate is mostly saved in .FIT binary files. The GARMIN connect web site enables a user to login and visualize the data related to his/her recorded activities and health status. The GARMIN Connect Mobile Applica- tion is required to synchronize the data between the watch and the web site.

We have developed an Android application called WEnnerMobile. It enables a user to view his activity history, visualize his heart rate records by curve chart, and test his/her pressure level by a answering a questionnaire. It will also provide a virtual coach to give the user personal advice on exercises. The user can choose a FIT or CSV fitness file stored in the phone memory. When a FIT file is chosen, it is be decoded into a CSV file, and the data can be processed and displayed in tables and charts.

Finally a website is used (WEnnerWeb) to store data collected with the watch and the mobile phone. The administrator of this web site can define the sport types, user's initial preferences of the sport and the target training time. Each time when an activity is finished, the WEnnerWatch app can collect Ecological Momentary Assessment and ask the user to self report on her mood (e.g. "Positive", "Negative" or "Neutral"). Such information can be exploited to tailor and adapt follow-up sessions.

<figure><img src="image_7.png" /><figcaption>Fig. 5. Devices and software architecture of the WEnner framework.</figcaption></figure>
# 3 Conclusions and Future Directions

We introduced the WE-nner technological framework for designing a coach that supports physical activity and enables personalized, multimodal and dynamic interactions between a user and a wearable. We explained and illustrated how this framework was implemented using a commercially available software development kit for smartwatches.

The next step is to define the specifications of the personalized multimodal coach. We will go through a requirement analysis involving users with different levels of physical activities practice. This requirement analysis will identify the desired functions of the coach as well as its multimodal interaction capacities. This includes the selection of opportune motivational interactions delivery timing [28].

Next steps include the modeling and implementation of relevant personality and inter-individual differences features in the WE-nner user's profile. We are considering two theories from Psychology: the OCEAN personality traits and the regulatory focus theory. These two theories because they have an impact either in terms of physical activity itself or in terms of persuasiveness. The OCEAN / Big Five personality traits, also known as the five factor model (FFM), is a model based on common language descriptors of personality [26]. The five factors have been defined as openness to experience, conscientiousness, extraversion, agreeableness, and neuroticism, often represented by the acronym OCEAN. Relations have been found between the OCEAN personality traits and motivation to learn [27], but also with physical exercise (see [21] for a review). For example, Tolea et al. found [20] found some associations between personality traits and physical activity level. Saklofske et al. observed that self-report emotional intelligence mediated the relationship between personality and exercise behavior [22].

The second theory that we are considering is the regulatory focus [24]. Regulatory focus has been shown to influence how individuals make judgments and decisions [23]. We have already regulatory focus in our MARC virtual agent platform [12, 13]. Regulatory focus is also being used for the generation of persuasive messages [11]. Individuals are either gain-oriented (“promotion-focused”) or loss-oriented (“prevention-focused”). Framing messages influence individuals’ cognitive processing of messages [25].

In terms of interaction, we will extend our framework for supporting the dynamic selection of output modalities and their combinations (e.g. complementarity, redundancy) to achieve an appropriate integration of the senses by users. This requires considering contextual information (e.g. if the user is on the move, a vibration can be used to inform the user that she should stop and look at an important message on the watch). Frameworks for multimodal output generation will be considered [19]. We will also consider the design of consistent interactions between the smartwatch, the mobile phone and a web site in order to support the relation between the user and her personalized coach which is in fact dispatched over several devices (possibly including other sensors and wearables).

We are also considering on how an animated and expressive agent displayed on the smartphone (and simple representations of it on the watch) can motivate the user based on affective reasoning and data [18].
These dimensions of personality and multimodal interaction will be considered as input for the modelling and implementation of the coach using the WEnner technological framework introduced in this paper.

Long term user studies need to be conducted to test if this wearable and its personalized and multimodal interactions do induce engagement and behavior change, and to assess how much they are impacted by aspect of the self [15, 16]. Ethics and privacy are also considered since personal data is collected and can induce behavior change.

Finally, sport coaching will be articulated with another coach in charge of stress management so that both coaches can exchange information in order to fine-tune and tailor consistent interactions.

## References

1. Ekkekakis, P.: Routeledge Handbook of Physical Activity and Mental Health. Routeledge Handbooks, New York (2013).

2. De Vries, P.W.: Persuasive Technology: Development and implementation of personalized technologies to change attitudes and behaviors. Adjunct Proceedings of the 12th International Conference, PERSUASIVE 2017, Amsterdam, The Netherlands, April 4-6, 2017.

3. de Vries, R. A. J., Truong, K.P., Zaga, C., Li, J., Evers, V.: A Word of Advice: How to Tailor Motivational Text Messages Based on Behavior Change Theory to Personality and Gender. Personal and Ubiquitous Computing. pp 1-13 (2017).

4. de Vries, R. A. J., Truong, K.P., Kwint, S., Drossaert, C., Evers, V.: Crowd-Designed Motivation: Motivational Messages for Exercise Adherence Based on Behavior Change Theory. In: Proceedings of the 2016 CHI Conference (2016).

5. Sano, A., P. Johns, M. Czerwinski, HealthAware: An Advice System for Stress, Sleep, Diet and Exercise. In: Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII), Xi'an, China (2015).

6. Bickmore, T.: Context Awareness in a Handheld Exercise Agent. Pervasive and Mobile Computing special issue on Pervasive Health and Wellness, Vol 5, 226-235 (2009).

7. Bickmore, T. W., Silliman, R. A., Nelson, K., Cheng, D. M., Winter, M., Henault, L., Paasche-Orlow, M.K.: A Randomized Controlled Trial of an Automated Exercise Coach for Older Adults. J. Am. Geriatr Soc., 61(10):1676-83 (2013).

8. Bjarne, M., Christian, R., Gutvik, C., Lavie, J., Nauman, J., Wisløff, U.: Personalized Activity Intelligence (PAI) for Prevention of Cardiovascular Disease and Promotion of Physical Activity. The American Journal of Medicine (2016).

9. Brinkman, P.: Virtual Health Agents for Behavior Change: Research Perspectives and Directions. In: Proceedings of the Workshop on Graphical and Robotic Embodied Agents for Therapeutic Systems (GREATS16) held during the International Conference on Intelligent Virtual Agents (IVA16), (2016).

10. Chen, J. J., Chung, Y.-F., Chang, C.-P., King, C. T., Hsu, C.-H.: A Wearable Virtual Coach for Marathon Beginners. In: Proceedings of the 20th IEEE International Conference on Parallel and Distributed Systems (ICPADS) (2014).

11. Corrégé, J.-B., Clavel, C., Sabouret, N., Hadoux, E., Hunter, A., Ammi, M.: Persuasive Dialogue System for Energy Conservation. In: Adjunct Proceedings of the 12th International Conference Persuasive Technology: Development and Implementation of Personalized Technologies to Change Attitudes and Behaviours (PERSUASIVE 2017), Amsterdam, The Netherlands, Peter W. de Vries, Thomas Van Rompay (Eds.), April 4-6, pp. 24-25 (2017).
12. Faur, C., Caillou, P., Martin, J.-C., Clavel, C.: A Socio-cognitive Approach to Personality: Machine-learned Game Strategies as Cues of Regulatory Focus. In: Proceedings of the 6th International Conference on Affective Computing and Intelligent Interaction (ACII 2015), Xi'an, China, pp. 581-587 (2015)

13. Faur, C., Martin, J.-C., Clavel, C.: Measuring Chronic Regulatory Focus with Proverbs: the Developmental and Psychometric Properties of a French Scale. Journal of Personality and Individual Differences. Vol. 107, pp. 137-145 (2017)

14. Lee, J., Lee, C., Jounghyun, G.K.: Vouch: Multimodal Touch-and-Voice Input for Smart Watches under Difficult Operating Conditions. Journal on Multimodal User Interfaces. Springer (2017).

15. Strachan, S. M., Whaley, D. E.: Identities, Schemas and Definitions: How Aspects of the Self Influence Exercise Behavior. In: Ekkekakis, P. (Ed.) Routeledge Handbook of Physical Activity and Mental Health. Routeledge Handbooks, New York, pp. 212-223 (2013).

16. McAuley, E., Mailey, E. L. Szabo, A. N., Gother, N.: Physical Activity and Personal Agency: Self-Efficacy as a Determinant, Consequence and Mediator. In: Ekkekakis, P. (Ed.) Routeledge Handbook of Physical Activity and Mental Health. Routeledge Handbooks, New York, pp. 224-235 (2013).

17. Noar, S. M., Benac, C. N., Harris, M. S.: Does Tailoring Matter? Meta-analytic Review of Tailored Print Health Behavior Change Interventions. Psychological Bulletin, 133(4), pp. 673-693 (2007)

18. Callejas, Z., Griol, D., McTear, M., Lopez-Cozar, R.: A Virtual Coach for Active Ageing Based on Sentient Computing and m-health. In: Proceedings of IWAAL, LNCS 8868, pp. 59-66 (2014)

19. Rousseau, C., Bellik, Y., Vernier, F., Bazalgette, D.: A Framework for the Intelligent Multimodal Presentation of Information. Signal Processing, Elsevier Publ., European Association for Signal Processing (EURASIP), ISSN: 0165-1684, Vol. 86, Issue 12, (2006)

20. Tolea, M.I., Terracciano, A., Simonsick, E.M., Metter E.J., Costa, Jr. P.T. and Ferruccic, L. Associations between personality traits, physical activity level, and muscle strength. J Res Pers. 2012 Jun; 46(3): 264-270.

21. Rhodes, R.E., Smith, N.E. Personality correlates of physical activity: a review and meta-analysis. Br J Sports Med. 2006 Dec;40(12):958-65.

22. Saklofske, D.H., Austin, E.J., Rohr, B.A., Andrews, J.J. Personality, emotional intelligence and exercise. J Health Psychol. 2007 Nov;12(6):937-48.

23. Cesario, J. et al.: Regulatory Fit and Persuasion: Basic Principles and Remaining Questions. Soc. Personal. Psychol. Compass. 2, 1, 444-463 (2008).

24. Higgins, E.T.: Beyond pleasure and pain. Am. Psychol. 52, 12, 1280 (1997).

25. Lee, A.Y., Aaker, J.L.: Bringing the Frame Into Focus: The Influence of Regulatory Fit on Processing Fluency and Persuasion. J. Pers. Soc. Psychol. 86, 2, 205-218 (2004).

26. McCrae, R.R., John, O.P. An introduction to the five-factor model and its applications. Journal of Personality. 60 (2): 175-215. 1992.

27. De Feyter, T., Caers, R., Vigna, C., Berings, D. Unraveling the impact of the Big Five personality traits on academic performance: The moderating and mediating effects of self-efficacy and academic motivation. Learning and Individual Differences. 22: 439-448. 2012.

28. Sano, A., Johns, P., Czerwinski, M. Designing Opportune Stress Intervention Delivery Timing using Multi-modal Data. International Conference on Affective Computing and Intelligent Interaction (ACII), San Antonio, Texas, October 2017.

29. Afzal, S. and Robinson, P. (2011). Designing for Automatic Affect Inference in Learning Environments. Educational Technology & Society, 14 (4), 21-34.
30. Bangert-Drowns, R. L., Kulik, C. C., Kulik, J. A., & Morgan, M. T. (1991). The instructional effect of feedback in test-like events. Review of Educational Research, 61, 213–238.

31. D' Mello, S., Graesser, A., & Picard, R. (2007). Toward an Affect-Sensitive AutoTutor. Intelligent Systems, IEEE, 22 (4), pp. 53–61.

32. Kim, K., Shin, D-H., Yoon, H. (2016). Information tailoring and framing in wearable health communication, Information Processing and Management 53 (2017) 351–358