# A VR-Based Serious Game Associated to EMG Signal Processing and Sensory Feedback for Upper Limb Prosthesis Training

Reidner Cavalcante, Aya Gaballa, John-John Cabibihan, Alcimar Soares, Edgard Lamounier

<figure><img src="image_1.png" /></figure>

## To cite this version:

Reidner Cavalcante, Aya Gaballa, John-John Cabibihan, Alcimar Soares, Edgard Lamounier. A VR-Based Serious Game Associated to EMG Signal Processing and Sensory Feedback for Upper Limb Prosthesis Training. 20th International Conference on Entertainment Computing (ICEC), Nov 2021, Coimbra, Portugal. pp.433-440, 10.1007/978-3-030-89394-1_36. hal-04144415

HAL Id: hal-04144415

https://inria.hal.science/hal-04144415

Submitted on 28 Jun 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Amnesia in the Atlantic: an AI Driven Serious Game on Marine Biodiversity

Mara Dionísio¹,²[0000-0003-0684-7512], Valentina Nisi¹,³[0000-0002-8051-3230], Jin Xin³, Paulo Bala¹,⁴[0000-0003-1743-0261], Stuart James⁵[0000-0002-2649-2133], and Nuno Jardim Nunes¹,³[0000-0002-2498-0643]

¹ ITI-LARSyS, Portugal {mara.dionisio,paulo.bala}@iti.larsys.pt

² Universidade da Madeira, Portugal

³ IST, Universidade de Lisboa, Portugal

{valentina.nisi,nunojnunes}@tecnico.ulisboa.pt

⁴ FCT, Universidade Nova de Lisboa, Portugal

⁵ Visual Geometry and Modelling (VGM) Lab & Pattern Analysis and Computer Vision (PAVIS), Istituto Italiano di Tecnologia, Italy stuart.james@iit.it

**Abstract.** The use of Conversational Interfaces has evolved rapidly in numerous fields; in particular, they are an interesting tool for Serious Games to leverage on. Conversational Interfaces can assist Serious Games' goals, namely in presenting knowledge through dialogue. With the global acknowledgment of the joint crisis in nature and climate change, it is essential to raise awareness to the fact that many ecosystems are being destroyed and that the biodiversity of our planet is at risk. Therefore in this paper, we present Amnesia in the Atlantic, a Serious Game enhanced with a Conversational Interface embracing the challenge of critically engaging players with marine biodiversity issues.

**Keywords:** Interactive Storytelling · Artificial Intelligence · Conversational Interfaces · Biodiversity · Nature Conservation.

## 1 Introduction

Over the years, we have witnessed that new Artificial Intelligence (AI) methods have become indispensable to orchestrate game dynamics. These range from the generation of storylines, game levels, enhanced graphics or to add intelligent behaviours to non-playable characters [5]. Conversational interfaces (CIs), also known as chatbots, are a form of AI created to have conversations with humans either by voice or by text [13]. This technology is becoming more and more intelligent and useful, and is implemented for multiple uses and contexts [1]. However, CIs have been an unexplored resource in games [7,16] even though they could be particularly relevant in Serious Games, where knowledge is passed through linguistic information in the form of spoken/written language [14]. Several studies highlight the positive impact of Serious Games in enhancing cognitive, skill-based, and affective abilities [17]. We are particularly interested in how Serious Games target the player's environmental consciousness [4]; making intangible
environmental issues more recognizable and aligned to players’ experience and daily lives [9]. For marine environmental issues, this is a barrier since it often explores issues they have no knowledge of and underwater spaces that are not accessible. To overcome this, there is a need to engage players with novel and creative interfaces and gameplay strategies [9]. This is why in this paper, we explore the design space of incorporating CIs in Serious Games that engage users in nature conservation [2]. Interactivity through dialogue enhances text-based learning [17] and incorporating natural language interfaces is an active topic of interest [7]. With the use of CIs, we intend for players to be able to critically engage with the issues and recognize their role in the ecosystem. This paper contributes with the design of *Amnesia in the Atlantic (AmA)*, a Serious Game for marine environmental awareness, incorporating CIs as a conduit for knowledge of megafauna biodiversity of the Macaronesia region. Through the description of the design of the artefact, and the design decisions behind it, we delineate future directions of research for our artefact.

# 2 Related Work

## 2.1 Marine Environmental Conservation in Serious Games

The entertainment industry is engaging in increasing people’s awareness regarding complex social issues, such as climate change and biodiversity conservation [15]. Benefits stemming from technological mediation of nature conservation include the engagement of larger audiences with the digital artifacts and their exposure to pro-environmental messages [6].

In particular, for marine biodiversity conservation, there are several examples of games [8] that aim at making users feel involved in restoring marine ecosystems by solving puzzles. In the narrative game *Beyond Blue* [19], the player takes on the role of a diver, that spends most of the time diving underwater and researching about marine creatures. The game narrative is used as an educational source to teach the players about marine biology. The *Ocean Rift* game [18], unlike the previous games, does not rely on puzzles or specific tasks to complete the game, but is rather an exploratory game of 12 underwater habitats, filled with a variety of animals and information about them [18]. These projects leverage immersion in the underwater world for player engagement. However, there is an opportunity to place the player in the role of the marine fauna, leveraging character persuasion and identification [12,11] to develop an emotional bond between the players and the marine species; this would allow for a better understanding of how ecosystem issues affect these animals. To accomplish this, we look at how CIs can add a new level of interactivity in the antromorphization of characters.

## 2.2 Conversational Interfaces in Serious Games

Conversational interfaces (CIs), have become popular as a means to create engaging human-computer interactions as they promise new ways to offer highly
interactive and personalized experiences [1]. For example, social chatbots such as Replika [13] can have deep conversations with the user while visual chatbots are created to have a conversation through the interpretation of images given by the user [3]. Since we are interested in CIs applied in the context of Serious Games, the work of Kowald et al. [10] is particularly relevant as it showcases three applications of CIs especially designed for learning through serious interactive fiction games. By analysing, three scenarios, the authors highlight the importance of having a clear idea of the specific application, the goals, and the target audience. Therefore, is vital to follow a good design process to ensure the quality of the CIs, and safeguard users' expectations against CIs' limitations. More recently, Göbl et al. [7] performed a literature review on CIs in Serious Games, where they analyse 30 works (from a broad range of applications domains) to understand the potential and future research directions of CIs in game-based learning. Their findings highlight an increased motivation and learning through social agents in the form of faces or embodied agents that also add non-verbal cues [7]. However, their research also exposes several limitations of CIs in Serious Games, such as users who initially showed engagement with the CIs, quickly would complain and give up when expectations are not met. Furthermore, the use of CIs in Serious Games is limited to what can be made with the available technical resources, budget, and of course, whether data-driven implementations can be built on sufficiently large data sets [7]. The literature in CIs applied to Serious Games is still quite limited, and there is still much do to in terms of design and evaluation, in order to derive general guidelines for future practitioners. Therefore, our goal is to contribute with the design of AmA to expand the body of work of Serious Games incorporating CIs, in particular, applied to the Marine Environmental Conservation cause.

## 3 Amnesia in the Atlantic

AmA, was inspired by Who Am I?, a guessing game where players use yes/no questions to guess the identity of a famous person or animal. In AmA, the players learn about the marine biodiversity of the Macaronesia ecosystem by trying to discover the characteristics of the animal that they are embodying. The game follows the simple narrative of a cetacean (aquatic mammal), that after being struck by a boat, is left wandering the Atlantic, affected by amnesia and in order to survive, needs to discover who they are. In each game, the player is randomly assigned to one of six species of cetaceans, and by exploring an underwater environment, representative of its marine ecosystem, the player has to uncover clues by talking to CIs, that lead to the discovery of who they are. The more they interact with the CI, the more they will know about themselves but also what kind of relationship they have in the marine ecosystem (e.g., feeding habits, symbiotic relationships, etc.). Looking into further details regarding the game play, Fig. 1 represents the interface flow of the game experience. After an initial 2D animation explaining the game goal, the player meets a non-playable character that will serve as a companion and advisor. This character, a turtle,
<figure><img src="image_4.png" /><figcaption>**Fig. 1.** Interface flow: while exploring environments (A), the player is introduced to the tutorial (B). Interacting with a species in the environment can establish a conversation (C). The game supports continued guessing (D) and a reward for the correct guess (E).</figcaption></figure>

is one of the most common species in the Macaronesia region. This first inter-
action serves as a tutorial, as this friendly turtle goes over the actions available
for the player, namely, navigation (in the 3D environment, and through a map
interface), dialogue (by approaching an animal and asking questions) and how
the player can guess their identity. After this initial tutorial, the player can
explore different locations. These locations are showcased in the map and this
exploration is non-linear. Currently, we envisage three locations with 3D envi-
ronments expressive of the Macaronesia region, purposely representing different
ocean zones to depict different ecosystems with different visual characteristics
(e.g. orography, flora, and lighting). For example, a littoral environment near the
coast will present different species than an environment in the deep sea. In this
way, the player can learn about the Macaronesia underwater environment simply
by space exploration. The characters in these environments are conversational
agents, possessing character traits consistent with the species they are repre-
senting. By interacting with these species and through question formulation, the
players will uncover clues about their identity. For example, this information can
include clues about how they look like, how fast they swim, feeding habits or
environmental threats that they might be facing. Once the player has retrieved
enough information, they can try to guess their identity by clicking on the bell
icon at the top right side of the screen. In case the player succeeds, more in-
formation about the species will be presented in a 3D exploratory view of the
animal they were embodying. If they are not able to guess, then they can go
back to the underwater environment to talk with more species (CIs).

AmA was designed by focusing on three pillars, (1) immersive environment, (2) characters, and (3) AI driven generative game play. Firstly, the immersive environments are intended to provide a realistic simulation of environments that are not accessible to players as humans. They also provide coded knowledge in the way they are visually presented and populated. Secondly, by embodying a
marine species and playing in a first person view, we intend to dissociate the
player from the experience of having a human body, while still being able to
have character identification through role-playing. Moreover, the partial antro-
morphization of the characters as capable of dialogue is intended to allow char-
acter persuasion and identification. Thirdly, the random allocation of a character
and the AI mechanic of conversational dialogue allows for generative game play
promoting replayability. To fully understand the dynamics between cetaceans in
the Macaronesia region, there are several species to discover and after finishing
a game, players can go back to play as another species. The game also offers a
personalised experience to each player, since the conversation with the marine
species will be different from player to player.

4 Conclusion and Future Work

In this work in progress paper, we presented AmA, a Serious Game enhanced
with a CI embracing the challenge of critically engage players with the marine
biodiversity issues. AmA will be developed in Unity<fnref n="6" /> for desktop computers,
which allows for the creation of realistic simulation and easy input from mouse
and keyboard devices. Dialogue with non-playable characters will be accom-
plished through a web-service (REST API) providing access to a question and
answer agent. The separation allows the agent to run on independent computa-
tional resources for scalability and stability. The agent<fnref n="7" /> will inspect a provided
image of marine animal and attempt to answer questions about it.

While we develop a functioning prototype for *AmA*, a mid fidelity proto-
type will be tested with participants to uncover early design limitations and to
understand dialogue within CIs in a game setting. By building on user's expec-
tations, experience and tolerance to failure when interacting with CIs, we intend
to understand how to prevail over the limitations of CIs and limit their impact
on the overall game experience. In the long term, we plan to test the prototype
to understand its impact in marine conservation awareness through the use of
pro-environmental behaviour scales. With *AmA*, we contribute to the body of
work in CIs used in Serious Games.

**Acknowledgements** This work was supported by the project INTERTAGUA -MAC2/1.1a/385. We also wish to acknowledge the work of our fellow researcher Rui Trindade.

References

1. Adamopoulou, E., Moussiades, L.: An Overview of Chatbot Technology. In: AI Applications and Innovations, vol. 584, pp. 373–383. Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-49186-4_31

<footnote n="6">⁶ Unity game engine - https://unity.com/</footnote>

<footnote n="7">⁷ Visual Question and Answering - http://vqa.cloudcv.org/</footnote>
2. Büscher, B.: Nature 2.0: Exploring and theorizing the links between new media and nature conservation. New Media & Society **18**(5), 726–743 (2016)

3. Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J.M.F., Parikh, D., Batra, D.: Visual dialog (2016)

4. Fjællingsdal, K.S., Klöckner, C.A.: Gaming green: The educational potential of eco – a digital simulated ecosystem. Frontiers in Psychology **10**, 2846 (2019). https://doi.org/10.3389/fpsyg.2019.02846

5. García-Sánchez, P.: Artificial Intelligence and Games. Genetic Programming and Evolvable Machines **20**(1), 143–145 (Mar 2019). https://doi.org/10.1007/s10710-018-9337-0

6. Gaver, W., et al.: My naturewatch camera: Disseminating practice research with a cheap and easy diy design. In: Proceedings of the CHI 2019. pp. 1–13 (2019)

7. Göbl, B., Kriglstein, S., Hlavacs, H.: Conversational Interfaces in Serious Games: Identifying Potentials and Future Research Directions based on a Systematic Literature Review: In: Proceedings of CSEDU 2021. pp. 108–115 (2021). https://doi.org/10.5220/0010447301080115

8. Jepson, P., Ladle, R.J.: Nature apps: Waiting for the revolution. Ambio **44**(8), 827–832 (2015)

9. Klöckner, C.A.: The Psychology of Pro-Environmental Communication. Palgrave Macmillan UK, London (2015). https://doi.org/10.1057/9781137348326

10. Kowald, C., Bruns, B.: New Learning Scenarios with Chatbots – Conversational Learning with Jix: from Digital Tutors to Serious Interactive Fiction Games. International Journal of Advanced Corporate Learning (iJAC) **12**(2), 59–62 (Nov 2019), number: 2

11. Minich, M.: The effects of narrative transportation and character identification on persuasion in the medium of comics. Ph.D. thesis (Jan 2017). https://doi.org/10.13140/RG.2.2.20322.79047

12. Moyer-Gusé, E.: Toward a Theory of Entertainment Persuasion: Explaining the Persuasive Effects of Entertainment-Education Messages. Communication Theory **18**(3), 407–425 (Aug 2008). https://doi.org/10.1111/j.1468-2885.2008.00328.x

13. Nima, N., Lee, T., Molloy, D.: Being friends with yourself: How friendship is programmed within the ai-based socialbot replika. Masters of Media (2017)

14. Picca, D., Jaccard, D., Eberlé, G.: Natural Language Processing in Serious Games: A state of the art. International Journal of Serious Games **2**(3) (Sep 2015). https://doi.org/10.17083/ijsg.v2i3.87

15. Rands, M.R., Adams, W.M., Bennun, L., Butchart, S.H., Clements, A., Coomes, D., Entwistle, A., Hodge, I., Kapos, V., Scharlemann, J.P., et al.: Biodiversity conservation: challenges beyond 2010. science **329**(5997), 1298–1303 (2010)

16. van Rosmalen, P., Eikelboom, J., Bloemers, E., van Winzum, K., Spronck, P.H.M.: Towards a game-chatbot: Extending the interaction in serious games. In: Felicia, P. (ed.) Proceedings of the 6th European Conference on Game-based Learning, Cork, Ireland. pp. 525–532. Academic Publishing International Ltd. (2012)

17. Stephanidis, C., et al.: Seven HCI Grand Challenges. International Journal of Human-Computer Interaction **35**(14), 1229–1269 (Aug 2019). https://doi.org/10.1080/10447318.2019.1619259

18. Takahashi, D.: The Ocean Rift in virtual reality is like relaxing in an IMAX theater - venturebeat, http://venturebeat.com/2016/04/09/the-ocean-rift-in-virtual-reality-is-like-relaxing-in-an-imax-theater/

19. Watts, R.: Beyond blue review a contemplative and educational diving sim. PC Gaming (6 2020)