Student's Attention Improvement Supported by
Physiological Measurements Analysis

Andreia Artífice, Fernando Ferreira, Elsa Marcelino-Jesus, João Sarraipa,
Ricardo Jardim-Gonçalves

► To cite this version:

Andreia Artífice, Fernando Ferreira, Elsa Marcelino-Jesus, João Sarraipa, Ricardo Jardim-Gonçalves. Student's Attention Improvement Supported by Physiological Measurements Analysis. 8th Doctoral Conference on Computing, Electrical and Industrial Systems (DoCEIS), May 2017, Costa de Caparica, Portugal. pp.93-102, 10.1007/978-3-319-56077-9_8. hal-01629591

HAL Id: hal-01629591

https://inria.hal.science/hal-01629591

Submitted on 6 Nov 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Student's Attention Improvement Supported by Physiological Measurements Analysis

Andreia Artífice, Fernando Ferreira, Elsa Marcelino-Jesus, João Sarraipa and
Ricardo Jardim-Gonçalves

CTS, UNINOVA, DEE/FCT-Universidade Nova de Lisboa, Portugal
{afva, flf, ej, jfss, rg}}@uninova.pt

**Abstract.** The focus of the most recent theories of emotional state analysis is the Autonomic Nervous System. Those theories propose that sympathetic and parasympathetic nervous systems interact antagonistically accordingly to each emotional state implying variations of interbeat intervals of consecutive heart beats. Emotional arousal and attention can be inferred based on the electrocardiogram (ECG) specifically through Heart Rate Variability (HRV) analysis, including the Low Frequency (LF), High Frequency (HF), and ratio LF/HF. The aim of this study is to analyze the impact of classic background music, in students' emotional arousal and attention, and performance in the context of e-Learning training courses. As a result, it is foreseen the development of a system integrating wearables to smoothly gather the mentioned biosignals, which will be able to sense user's emotions to further automatically propose recommendations for better learning approaches and contents, aiming student's attention improvement.

**Keywords:** eLearning, Heart Rate Variability, emotional arousal, attention.

## 1 Introduction

The present research work analyses the role of classic background music in the students' attention, emotional arousal, and performance in the e-Learning environment. The proposed approach is based on ECG readings with the aim of feeding a smart system able to perform analysis and propose recommendations for improved learning approaches.

In the context of Internet based teaching, E-learning is becoming an innovative approach for delivering well-designed, learner-centered, interactive methodology. In that context E-learning is providing a learning environment to anyone, anyplace, anytime by utilizing the attributes and resources of various digital technologies, and for that it is growing its importance in the market [1]. The near future of E-learning will, most probably, be supported by the integration of sensing, diagnose, describe and qualify a given situation. Such exercises will be related with the specification of its related Smart Systems. The aimed characteristic of that Smart Systems would be the potential to sense using biosensors able to measure various physiological
parameters such as heart rate (HR), galvanic skin response (GSR), electroencephalogram (EEG) and electrocardiogram (ECG) [2, 3].

The hereby-presented research work follows the traditional research method and encloses the following research question: Does eLearning training courses with classic background music influence emotional states and will it have an impact in attention and performance?

Concerning the proposed research question, aforementioned, we argue, by hypothesis that if we define a prototype, able to conduct analysis of arousal and attention based on biosignals, then the process to determine the emotional arousal and attention of learning students could be seamlessly executed. Such development would be able to proof the usefulness of background music in eLearning materials.

This paper is organized as follows: next section is dedicated to Smart Systems and e-Learning. The state of the art, concerning background music and Emotional State and Attention detection using Electrocardiogram is presented in section 3, followed by the experiment in section 4. Section 5 reveals the results and finally section 6 presents the conclusions of the study.

## 2 Smart Systems and e-Learning

Smart systems are understood as systems able to sense, diagnose, describe and qualify a given situation as well as be capable to mutually address and identify each other. They are also able to interface, interact and communicate with their environment and with other smart systems [4]. Sensors and actuators are part of the Smart Systems Ecosystem that will become, most of the times, its central component. Different approaches have been proposed in the context of E-learning, among those we highlight the development of systems based on context and emotional sensing.

Zarrad et al. [5] propose a smart system with an architectural approach in which modalities are selected dynamically based on the operating context. The system considers context from different features including lighting, noise level, network connectivity, communication costs, communication bandwidth, and the social and medical situations.

Emotions have an important role in learning, and are significantly related with students' motivation, learning strategies, cognitive resources, self-regulation and academic achievements [6]. Several contributions were developed based on the identification of a user's emotional state with the goal of improving the eLearning experience. The Framework for Improving Learning Through Webcam's And Microphones (FILTWAM) [7] was developed with the aim of interpreting the emotional state of users in an e-learning environment by using webcam's and microphone's. This study focused only on facial mood detection and presents a structured framework composed by the following layers: learner, device, application and data with the most complex application layer on top of it and including an affective computing tool. In that same direction, Santos et al. [8] conducted a research in computer assisted language learning, using an Arduino based infrastructure that could sense the learners' affective state and deliver interactive recommendations. For that it was developed the Ambient Intelligence Context-aware Affective
Recommender Platform (AICARP), a platform similar to the one that will be
developed in the context of this study. A particularity of the platform is the sensor
module that receives biosignals from the learner and information from the
environment and a recommendation module that takes advantage of the intelligence of
the environment using internal and/or external actuators to offer interactive
personalized support to the learner. This overview brings the relevant smart systems
related to the work developed in the scope of the proposed research question. In
particular we look to design a smart system that analyses data from bodily-connected
sensors, analyses such data and by reasoning over selected algorithms identifies a
person's emotional state. Such systems, while connected with other similar smart
systems, can produce markers that give indications of common properties of different
people while listening to music and performing a learning task (e.g. attending an E-
learning course).

3 State of the Art

## 3.1 Background Music

The "Mozart effect", i.e. the scientific finding that listening to Mozart's piano sonata enhanced spatial reasoning [9], was supported by McCuntheon [10], while other studies found no support [11, 12]. Despite that, the study conducted to new research and subsequent hypothesis in the field. A study conducted by Thompson et al. [13] suggested that the "Mozart effect" performs a sort of manipulation of arousal and mood that conducted to enhanced performance. According to Day et al. [14], the effect of background music is rather specific and thus, needs to be tested on a selected target situation [14].

Concerning computer based environments, it was visible that, when reading news
from a pocket computer in a distracting environment, the reading rate and efficiency
was improved in the presence of fast music than slow music [5]. A study by
Thompson et al. [13] revealed that background music disrupted the ability of writing
 fluently.

In order to study the effect of background music in virtual environments for
learning Richards et al. [15], it was created a specific environment for a history
lesson. Results revealed that listening to the soundtrack of ‘Oblivion’ conducted to an
improvement in the number of facts remembered by the participants.

It is important to notice that the majority of the reported findings about background
music are not representative of computer-based environments. Thus, considering the
actual impact of technology in daily life it is crucial to continue progressing on that
specific research field.
## 3.2 Emotional State and Attention Detection Using Electrocardiogram

Physiological measurements, along with face recognition, or other measurements (e.g. posture and body movements) are the indirect clues about a person's emotional states since those cannot be assessed with some sort of direct measurements. Because of that, biosignals, such as electrocardiogram (ECG), electromyogram (EMG), electroencephalogram (EEG), galvanic skin response (GSR), blood volume pressure (BVP), heart rate (HR) or heart rate variability (HRV), temperature (T), respiration rate (RR), become the most important vital signals of the human body for an emotional assessment. Although the majority of the studies use different biosignals in conjunction with ECG in order to detect the emotional states [16, 17, 18], there are studies focused only in relating ECG with the expression of human emotions [19, 3].

Emotions and Autonomic Nervous System (ANS) are interrelated. ANS activity is viewed as the major component of the emotions in many recent theories about emotional assessment [20]. ANS is composed by the sympathetic and parasympathetic nervous system components [20]. The sympathetic nervous system is stimulated by changes in external environment, reacting mostly to any situation perceived as an emergency [20]. The primary function of parasympathetic nervous system is for example to coordinate activities related to the restoration and conservation of the body [21]. Both systems interact antagonistically causing the variations in interbeat intervals of consecutive heart beats [22]. Parasympathetic nervous system is more active in calm periods decreasing heart rate (HR) [17, 23]. However sympathetic activity is predominant when arousal increases conducting to the acceleration of HR [17, 23].

HRV is a measure of the continuous interplay between sympathetic and parasympathetic influences on heart rate that yields information about autonomic flexibility and thereby represents the capacity for regulated emotional response [22].

Several HRV parameters can be measured resulting from time domain analysis or frequency domain analysis. Among those parameter is the ratio between LF/HF, which is known as the active indicator of the sympathetic nervous system activity. High values of LF/HF correspond to the predominance of sympathetic activity and thus high arousal in opposition to parasympathetic activity for which the ratio LF/HF is lower, corresponding to a low arousal [24].

Progressing further towards the relevance of such parameters for the current study, it is important to value the physiological signals from ECG as a good source to analyze and predict the lack or presence of cognitive attention in individuals during task execution [22] that can achieve similar performance than EEG. Scientific evidence shows that there is a close relation between variability in heart rate and attention in adults [25], infants [26] and children [27]. It also has been shown that during tasks that require attention, HF and LF parameters of HRV have a decrease [28].
## 4 Experiment

The proposed experiment was designed to study emotional arousal and attention in the context of eLearning and was prepared as follows. Ten individuals where enrolled in the study (1 female and 9 males). Their ages were between 32 and 47. The participants have different educational backgrounds, all completed at least high school, and none was employed in the aquaculture business. This fact was not considered relevant since the aim was not the course contents but it was considered of relevance as they would be similar to an aquaculture candidate willing to have training in such matters.

The experiment was conducted in the scope of the AquaSmart project [29], which has the focus on data mining and big data technologies applied to the aquaculture sector. AquaSmart helps companies to transform raw data into knowledge via accurate business-driven analytical models in a seamless and efficient process. The AquaSmart project has a component of training and that leads to the relevance of such experiment in the scope of the project's training framework.

The materials for the execution of the experiment were adopted from AquaSmart training program and adapted for the hereby described experimental setup. The study comprises two e-learning courses: a) Introduction to AquaSmart (course 1); b) History of aquaculture (course 2). The second course was delivered with an extract from the beginning of the first part of Bach's Brandenburg Concerto Nº 2 as background music, which was rated as expressing happiness [30]. Both courses consisted of a learning component composed by 4 slides followed by a test with 5 questions about the learned materials. Previous studies already evaluated the impact of music in a person's mood, in this case it is aimed a step further in evaluating music's impact on attention [31].

During the sessions ECG measurements were executed using a Olimex SHIELD-EKG-EMG. In the end of the session a questionnaire was performed to quantify: a) the attention of the participants in each training course; b) the likability of each course; c) the difficulty of each course.

The sample was divided into two groups (Figure 1): a) a group that performed the training course 1 with no music followed by the training course 2 with music in an automatic form, i.e., with a pre-determined time interval between slides; b) a group that performed the training course 1 with no music followed by the training course 2 with music in an interactive form, i.e., requiring by the user to press the “Next” button of the interface to move to the next slide.
<figure><img src="image_2.png" /><figcaption>Fig. 1. Experimental design.</figcaption></figure>

The participants were informed about the procedures and while agreeing each signed a consent form. The procedural execution started by installing clinical electrodes to his/her left arm, right arm and right leg. Heart rate was measured 30 seconds before the start of each e-learning training course for baseline purposes and data was collected during the attendance to the e-learning courses. After the session, each subject rated the level of difficulty, attention and motivation towards each training course on a seven-point scale instantiated from “very little” (1) to “very much” (7).

Heart Rate Variability was measured, according to specification, as beat to beat variation in the heart rate (R-R intervals) of ECG. Such measurement quantifies the interplay between sympathetic and parasympathetic system of ANS, and it was acquired using Artifact software [2]. Special emphasis was given to LF and HF of HRV, related with attention, and the ratio LF/HF indicator of arousal (more details in Section 3.2) that were measured while resting and along the proposed learning tasks.

## 5. Results

As an example, the results of participant 1 (P1), concerning training course “AquaSmart” (course 1) with no music, revealed that LF/HF ratio increased during learning phase, as expected. Since LF and HF values, calculated in milliseconds squared, decreased; it can support that such participant was more attentive during the learning stage (Table 1). The performance achieved during the test phase for training course 1 in an automatic form was 80%.

The next tables present LF and HF measurements for Course 1 (no music) and Course 2 (with music) respectively for the same person designed as P1.

<table><caption>Table 1. Training Course 1 with no Music for P1.</caption><thead><tr><td></td><td>Resting</td><td>Learning</td><td>Test</td></tr></thead><tbody><tr><td>LF (ms<sup>2</sup>)</td><td>8074.65</td><td>4899.53</td><td>1453.03</td></tr><tr><td>HF (ms<sup>2</sup>)</td><td>6336.96</td><td>2479.01</td><td>4787.75</td></tr><tr><td>LF/HF</td><td>1.27</td><td>1.98</td><td>0.95</td></tr></tbody></table>
<table><caption>**Table 2.** Training Course 2 with music for P1.</caption><thead><tr><th></th><th>Resting</th><th>Learning</th><th>Test</th></tr></thead><tbody><tr><td>LF (ms²)</td><td>7377.99</td><td>2868.48</td><td>2996.58</td></tr><tr><td>HF (ms²)</td><td>4995.74</td><td>2761.50</td><td>2341.79</td></tr><tr><td>LF/HF</td><td>1.48</td><td>1.04</td><td>1.28</td></tr></tbody></table>

Participant P1 commented, after performing training course “History of Aquaculture” (course 2) with music, that the volume was excessively high and that he usually does not appreciate history related contents. Those remarks could explain the reason why HRV analysis revealed a decrease in the ratio LF/HF during learning phase (Table 2), i.e., the low level for emotional arousal indicator meaning that he didn’t like the contents of the course. Parameters related with attention (LF and HF absolute values) decreased (table 2) revealing a high level of attention while comparing to the other course stages. The performance achieved by this participant during test phase in course 2, in an automatic form was 60%.

Additionally, from the collected information of all questionnaires participants it was concluded that some of them like to study with music and others not. From the data collected using the ECG, it was possible to infer the same. This means that music helps in the learning and test phases only of a part of the participants. Such conclusion is in line to the study made by of Gião et al. in [31], which says that a same music can result in different emotional states to each person. This is also in line to the standard deviation of the graph of Figure 2, where with music (Figure 2 – right graph) it has a σ= 4.67 and without music (Figure 2 – left graph) it has a σ= 2.48. This means that there is more variability of values in the first case (course with music) representing its tendency for amplified results, which implies also an amplified arousal (emotional intensity). That means that people feel the music reacting accordingly, independently if in a positive or negative way. In conclusion could be said that despite being in a positive or negative way, classic background music enhances intensity of emotion in eLearning experiences.

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** At Left) Average LF/HF ratio for course 1 with no music presented in both automatic and interactive way; At Right) Average LF/HF ratio for course 2 with music presented also in both ways.</figcaption></figure>

When analyzed the engagement of students in the courses with music in relation to the one without can be verified that a higher commitment attention it is obtained in
courses with music. That can be verified in Figure 3 by lower average LF and HF values in absolute terms in courses with music.

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** At Left) Average LF(ms²) and HF (ms²) for course 1 with no music; At Right) Average LF (ms²) and HF (ms²) for course 2 with music.</figcaption></figure>

Figure 4 presents the average LF and HF values of all the students, from which could be inferred that students were more attentive or engaged in the automatic course. This means, that since they had not the possibility of going back or forward to see the contents again. It indicates that in order do not fail in the test they tried to be more attentive to the contents. This is explained by the reducing in time of the LF and HF absolute values. The graph in the left of Figure 4 shows that in relation to the “automatic” course. On the other hand, the graph in the right part of Figure 4 shows the opposite.

<figure><img src="image_5.png" /><figcaption>**Fig. 4.** At left) Average LF (ms²) and average HF (ms²), for courses presented in an automatic form; At right) Average HF (ms²) and average LF (ms²) for courses presented in an interactive form.</figcaption></figure>

**6. Conclusions and Future Work**

The use of wearables able to obtain LF and HF values from an ECG potentiates the creation of smart systems. This can be applied to the area of education, which will allow the system to sense the students’ physiological signals enabling the possibility of inferring particular situations about the level of the attention of the student.
Through this kind of systems could be implemented specific recommendation
services that will suggest particular contents or topics to recover the students
attention.

A smart system prototype able to conduct analysis of arousal and attention based on biosignals using wearables to measure ECG is proposed with incorporation of the analyzed HRV parameters. Thus, the process to determine the emotional arousal and attention of learning students can be facilitated and recommendations, such as a pause in the study, adjust the volume, presence or absence of the music could be suggested by the smart system.

As future work would be interesting to apply the study to a numerous population in order to have more reliable results since interesting results were found in the presented experimental setup.

**Acknowledgements.** The authors acknowledge the European Commission for its support and partial funding and the partners of the research projects from ERASMUS+: Higher Education – International Capacity Building - ACACIA – Project reference number – 561754-EPP-1-2015-1-CO-EPKA2-CBHE-JP, (http://acacia.digital); and Horizon2020 - AquaSmart – Aquaculture Smart and Open Data Analytics as a Service, project number - 644715, (http://www.aquasmartdata.eu/).

References

1. B. Khan: Managing E-Learning: Design, Delivery, Implementation and Evaluation. Information Science Publishing (2005)

2. Kaufmann, T., Sütterlin, S., Schulz, S.M. and Vögele, C.: ARTiiFACT: a tool for heart rate artifact processing and heart rate variability analysis. Behavior research methods, 43(4), pp.1161--1170 (2011).

3. Appelhans, B. M., Luecken, L. J.: "Heart Rate Variability as an Index of Regulated Emotional Responding", Review of General Psychology, 10(3), 229—240 (2006).

4. European Commission ICT Work Programme 2007-08

5. Zarrad, A.; Zaguia, A.; Building a dynamic context aware approach for smart e-learning system. In: Computing Technology and Information Management (ICCTIM), 2015 Second International Conferen, pp. 144--149. IEEE (2015)

6. Pekrun, R., Goetz, T., Titz, W., Perry, R. P.: Academic Emotions in Students' Self-Regulated Learning and Achievement: A Program of Qualitative and Quantitative Research. Educational Psychologist, 37:2, pp. 91--105 (2002)

7. Bahreini, K., Nadolski, R. and Westera, W.: Towards multimodal emotion recognition in e-learning environments. Interactive Learning Environments, 24(3), pp.590--605 (2016)

8. Santos, O. C., Saneiro, M., Boticario, J. G., Rodriguez-Sanchez, M. C.: Toward interactive context-aware affective educational recommendations in computer-assisted language learning. New review of Hypermedia and Multimedia, 22(1-2), pp. 27—57 (2016)

9. Steele, K., Bass, K., Cook, M.: The mystery of the Mozart effect: failure to replicate, Psychological Science, 10(4), pp. 366--369 (1999)

10. McCuntheon, L.: Another failuree to generalize the Mozart effect, Pshychological Reports, 87(1), pp. 325--373 (2000)

11. Ransdell, S., Gilroy, L.: The effects of background music on word processed writing, Computers in Human Behavior, 17, pp. 141--148 (2001)
12. Kallinen, K.: Reading news from a pocket computer in a distracting environment: effects of the tempo of background music, Computers in Human Behavior, 18: 537--551 (2002)

13. Thompson, W. F., Schellenberg, E. G., Hussain, G.: Arousal, mood and the Mozart Effect, Pshychological Science, 12(3), pp. 248--251 (2001).

14. Wilson, T., Brown, T.: Reexamination of the effect of Mozart's music on spacial-task performance, The Journal of Psychology, 131(4), pp. 365-370 (1997)

15. Richards, D., Fassbender, E., Bilgin, A., Thompson, W. B.: An investigation of the role of background music in IVWs for learning, ALT-J 16(3), pp. 231--244 (2008)

16. Kim, J., André, E.: Emotion recognition based on physiological changes in music listening, IEEE Trans. Pattern Analysis and Machine Intel., 30(12): 2067—2083 (2008)

17. Schaaff, K., Adam, M. T. P: Measuring Emotional Arousal for Online Applications: Evaluation of Ultra-Short Term Heart Rate Variability Measures, Humaine Association Conference on Affective Computing and Intelligent Interaction, pp. 362—368 (2013)

18. Kreibig, S. D.“Autonomic nervous system activity in emotion: A review”, Biological Psychology, 84, pp. 394-421 (2010)

19. katsis, C.D., Kater, Ganiatsas, G., Fotiadis, D. I.: Towards Emotion Recognition in Car-Racing Drivers: A Biosignal Processing Approach, IEEE Transactions on Systems, Man, and Cybernetics. Part A: Systems and Humans, 38(3), (2008).

20. McCorry, L. K.; Physiology of the autonomic nervous system. American journal of pharmaceutical education, 71(4), 78, (2007)

21. Freeman, J. V.; Dewey, F. E., Hadley, D. M., Myers, J.; Froelicher, V. F; Autonomic nervous system interaction with the cardiovascular system during exercise. Progress in cardiovascular diseases, 48(5), 342-362, (2006)

22. Appelhans, B. M.; Luecken, L. J.; Heart rate variability as an index of regulated emotional responding. Review of general psychology, 10(3), 229, (2006)

23. Diffen; Parasympathetic vs. Sympathetic Nervous System. http://www.diffen.com/difference/Parasympathetic_nervous_system_vs_Sympathetic_ner vous_system [Accessed 04-Feb-2017]

24. Fruhata, T., Miyachi, T. and Adachi, T.: Doze driving prevention system by low frequency stimulation and high density oxygen with fragrance of GF (Grape Fruit). In: International Conference on Knowledge-Based and Intelligent Information and Engineering Systems, pp. 11--20. Springer Berlin Heidelberg (2011).

25. Porges, S.W., Raskin, D.C.: Respiratory an heart rate components of attention. J. Exp. Psychol. 81, 497—503 (1969).

26. Richards, J.E., Casey, B.J.: Heart rate variability during attention phases in young infants. Psychophysiological 28, 43-53 (1991)

27. Porges, S.W. Heart rate indices of newborn attentional responsivity. Merill-Palmer Quart. 20, 231--254 (1974)

28. Tripathi, K.K., Mukundan, C. and Mathew, T.L.: Attentional modulation of heart rate variability (HRV) during execution of PC based cognitive tasks. Ind J Aerospace Med, 47(1), pp.1--10 (2003)

29. AquaSmart; http://www.aquasmartdata.eu/ [Accessed 04-02-2017]

30. Kallinen, K.: Emotional ratings of music excerpts in the western art music repertoire and their self-organization in the Kohonen neural network. Psychology of Music, 33(4), pp.373--393 (2005)

31. Gião, J.; Sarraipa J.; Francisco-Xavier, F.; Ferreira, F.; and Jardim-Goncalves R.; Zdravković M.: Profiling Based on Music and Physiological State. In the 8th International Conference on Interoperability For Enterprise Systems And Applications (I-ESA 2016), held on Guimarães, Portugal on March 29-April 1, (2016)