# A Memory Game Proposal for Facial Expressions Recognition in Health Therapies

Samuel Vitorio Lima, Victor T. Sarinho

<figure><img src="image_1.png" /></figure>

## To cite this version:

Samuel Vitorio Lima, Victor T. Sarinho. A Memory Game Proposal for Facial Expressions Recognition in Health Therapies. 19th International Conference on Entertainment Computing (ICEC), Nov 2020, Xi'an, China. pp.264-269, 10.1007/978-3-030-65736-9_23. hal-03686005

HAL Id: hal-03686005

https://inria.hal.science/hal-03686005

Submitted on 2 Jun 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Memory Game Proposal for Facial Expressions Recognition in Health Therapies

Samuel Vitorio Lima¹, Victor Travassos Sarinho¹

Universidade Estadual de Feira de Santana (UEFS)
Laboratório de Entretenimento Digital Aplicado (LEnDA)
Feira de Santana, Bahia, Brazil
samvlima10@gmail.com, vsarinho@uefs.br

**Abstract.** The recognition of facial expressions has an important role in the communication and interaction between human beings. Researches attest that it is possible to exercise the recognition skills and production of facial expressions with the aid of computational support tools. This article presents Trimemória, a game whose objective is to produce a therapeutic playful environment for children with difficulty in recognizing facial expressions. Trimemória seeks to apply classic dynamics of memory games with sensors, in an attempt to increase the integration of the real world with the virtual interactions provided by the game.

**Keywords:** Health game · Facial expressions · RFID

## 1 Introduction

The recognition of facial expressions is an intrinsic ability of human beings [4], representing an important role in the communication and interaction between humans. However, some children can show some difficulties at the task of facial expressions recognition [1], such as Autism Spectrum Disorder (ASD) carriers.

The ASD is classified in the group of neurodevelopmental disorders [7], which result in significant losses in the communication, the interpersonal relationships, the imaginative ability and movement [8]. The ASD does not have a cure, but psychological approaches can be useful for the improvement of behavior of individual ASD carrier, given a life improvement and not allowing a worse condition [9].

Searches attest that it is possible to exercise the recognition abilities and production of facial expressions of ASD carriers, through the support of visual computing tools capable of reinforce the connection between mental state and facial expression [5, 11, 3]. In this sense, this article presents the development of a therapeutic support game for children, in special who carries ASD, called Trimemória. It is a game that extends the popular memory game with facial elements for expression recognition, together with *Radio Frequency IDentification* (RFID) resources to produce a real and virtual playful therapeutic ambient for the child audience.
## 2 Related Work

The RFID technology is used in the automated identification of people and objects. It is a sensor which extends the integration between real and virtual world in the smart environment [2]. As a result, RFID allows computer programs or cell phones to track changes in the physical world and reflect them in the virtual world [2].

Some developed initiatives has been applied with the use of RFID technology to the support of educational activities for children with ASD. As an example, in the United Kingdom (U.K.), teachers use the Pictures Exchange Communication System (PECS) for communication support with children who carries ASD. Based in the PECS, Herring et al. [6] developed a virtual teacher for the child interaction that recognizes images with embedded RFID tags, giving as a result a certain autonomy to children interact and learn with it.

Another example of RFID as a support tool was created by Sula and Spaho [10], where cards with RFID tags were used to support assistive education activities, such as images learning, identify equals object, coo-work, emotional categories, and lastly math. The main use of RFID in this system was performed by parents that can index some objects. When the child takes an indexed card and goes through the sensor, he will hear the word in the native language with the proposal to repeat it [10].

## 3 Methodology

Trimemória uses distinct memory cards with at least three images of the same creature with different facial expressions to be found by the player (Fig. 1). The idea is to allow distinct expressions identification in the same creature, expanding intervention possibilities during therapeutic sessions. To associate a virtual environment to the game, each Trimemória card has a built-in RFID tag that will be read by an integrated RFID service. The idea is to detect which creature the child found and generate an associated stimulus for it.

For each detected RFID tag, an information is sent to a MQTT broker that transfers the detection to connected clients. The action of sending the RFID tag is done by an RFID service supported by an RFID device, and each connected client represents an application that generates a virtual stimulus to the player for each received information.

Another point is the RFID service assembly with a low cost and *open hardware* micro-controller able to read RFID sensors. For this, a *NodeMCU Lolin 1.0* was used to: collect the data of an RFID sensor; manage the connection with the WiFi service; manage the connection with the provided broker; and send RFID tags detected by the protocol MQTT's topic.
<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Some designed creatures with respective emotions for the Trimemória game.</figcaption></figure>

The connection between NodeMCU and the RFID sensor is performed by a communication SPI (Serial Peripheral Interface). The information is transmitted via pins D4, D5, D6 and D7, which represents the *Reset* (RST), *Serial Clock* (SCK), *Master input Slave Output* (MISO) and *Master Output Slave Input* (MOSI), respectively (Fig. 2).

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Prototyping the RFID service for the Trimemória game.</figcaption></figure>
The MQTT server/broker is responsible for managing the messages of the topics, which contains the IDs of the detected RFID tags by the connected RFID service. After receiving a message, the MQTT server sends the message to all clients connected with the topic, in this case the *outPut* topic of the Trimemória game.

<figure><img src="image_5.png" /><figcaption>Fig. 3. RFID, MQTT and HTML services for the Trimemória game.</figcaption></figure>

A TCP server was also developed to be activated when the RFID service turns on. It is responsible to collect available WiFi information from an external dispositive (smartphone for example) that will be used by the RFID service to connect with the MQTT broker. After the confirmation that the internet information is available to be used, the RFID service turns off the TCP server and connects itself with the chosen WiFi, and consequently with the respective active MQTT broker (Fig. 3).

A HTML server was also created to act as a client to the MQTT broker (Fig. 3). It was developed through available JavaScript frameworks able to: interpret the developed MQTT client code (Node.js); manage the information from HTTP requisitions and provide an API REST (CORS and Express.js); perform the connection with the MQTT broker(MQTT.js); and create dynamic views for the game (Vue.js). As a result, when the MQTT broker sends a new information to the MQTT client to API REST, this information will appear in a connect client interface managed by Vue.js.
# 4 Results and Discussion

For the initial RFID service prototype (Fig. 4), that was built using a low cost open hardware (U$ 4.23 as total cost), it is able to execute two main processes for the game operation. The first is responsible to establish a communication to receive the necessary settings to control the communication with the MQTT broker. The second is responsible to control the communication with the MQTT broker, by sending detected tag IDs from the RFID device. In both, the game presented a good server performance and a fast device response, without compromise the game play as a whole.

<figure><img src="image_6.png" /><figcaption>**Fig. 4.** RFID service prototype, game cards with RFID tags, and an HTML client example.</figcaption></figure>

For the game cards with associated tag IDs, they will be detected by the RFID device when the player selects one. The RFID service sends a message via MQTT broker to the HTML server, that is received by the HTML client due to a continuous verification of the HTML server status (Fig. 3). As a final result, for each received tag ID by the MQTT broker, the associated images and sounds of the detected emotion is displayed to the player (Fig. 4).

# 5 Conclusions and Future Work

This paper presented Trimemória, a game that creates a mixed reality in a memory game, which can be used in health therapies to assist the patient in the recognition of facial expressions. For that, RFID, MQTT and HTML services were developed to provide: *hardware devices*, for player interaction on a physical level; *software services*, for the card detection together with internet communication; and *client apps*, able to present multimedia content associated with detected facial expression.

Regarding the use of open hardware, this paper confirms that its use is viable for the creation of an initial low-cost prototype of the proposed game, despite
the different types of developed and configured services for its proper functioning. As an example, the creation of an extra TCP application to establish the communication between the RFID service and the MQTT broker shows that it is necessary to automate and simplify the activation and the synchronization process of the proposed hardware/software resources for the game.

As future work, it is intended to create an approach for the direct communication between RFID and the virtual game interface, eliminating the use of different services applied in the developed prototype. Other game dynamics, such as the competitors identification in a match and the return of game awards points, will be developed in the future. To finish, the application of the game in therapies with ASD carriers, along with the assessment of gains and losses obtained with its use, will also be carried out in a near future.

## References

1. Baron-Cohen, S., Leslie, A.M., Frith, U., et al.: Does the autistic child have a "theory of mind". Cognition **21**(1), 37–46 (1985)

2. Bohn, J.: The smart jigsaw puzzle assistant: Using rfid technology for building augmented real-world games. In: Workshop on Gaming Applications in Pervasive Computing Environments at Pervasive. vol. 2004 (2004)

3. Deriso, D.M., Susskind, J., Tanaka, J., Winkielman, P., Herrington, J., Schultz, R., Bartlett, M.: Exploring the facial expression perception-production link using real-time automated facial expression recognition. In: European Conference on Computer Vision. pp. 270–279. Springer (2012)

4. Eckman, P., Friesen, W.: Manual for the facial action coding systems, consulting psych. Press, Palo Alto (CA) (1977)

5. Gordon, I., Pierce, M.D., Bartlett, M.S., Tanaka, J.W.: Training facial expression production in children on the autism spectrum. Journal of autism and developmental disorders **44**(10), 2486–2498 (2014)

6. Herring, P., Sheehy, K., Jones, R., Kear, K.: Designing a virtual teacher for non-verbal children with autism: Pedagogical affordances and the influence of teacher voice (2010)

7. Klin, A.: Autismo e síndrome de asperger: uma visão geral. Brazilian Journal of Psychiatry **28**, s3–s11 (2006)

8. Lô, E.N., Goerl, D.B.: Representação emocional de crianças autistas frente a um programa de intervenção motora aquática. Revista da graduação **3**(2) (2010)

9. Pradi, T., Silva, L., Bellon, O.R., Dória, G.M.: Ferramentas de computação visual para apoio ao treinamento de expressões faciais por autistas: Uma revisão de literatura. In: Anais do XLIII Seminário Integrado de Software e Hardware. pp. 140–151. SBC (2020)

10. Sula, A., Spaho, E.: Using assistive technologies in autism care centers to support children develop communication and language skills. a case study: Albania. Academic Journal of Interdisciplinary Studies **3**(1), 203–203 (2014)

11. Tanaka, J.W., Wolf, J.M., Klaiman, C., Koenig, K., Cockburn, J., Herlihy, L., Brown, C., Stahl, S., Kaiser, M.D., Schultz, R.T.: Using computerized games to teach face recognition skills to children with autism spectrum disorder: the let's face it! program. Journal of Child Psychology and Psychiatry **51**(8), 944–952 (2010)