# New Attitude to Learning in Virtual Environments - Mining Physiological Data for Automated Feedback

Zdena Lustigova, Aude Dufresne, François Courtemanche

► To cite this version:

Zdena Lustigova, Aude Dufresne, François Courtemanche. New Attitude to Learning in Virtual Environments - Mining Physiological Data for Automated Feedback. Second IFIP TC 13 Symposium on Human-Computer Interaction (HCIS)/ Held as Part of World Computer Congress (WCC), Sep 2010, Brisbane, Australia. pp.297-300, 10.1007/978-3-642-15231-3_34. hal-01055467

HAL Id: hal-01055467

https://inria.hal.science/hal-01055467

Submitted on 13 Aug 2014

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
New Attitude to Learning in Virtual Environments -
Mining Physiological Data for Automated Feedback

Zdena Lustigova ¹, Aude Dufresne ², François Courtemanche³

¹Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic

²Department of Communication, University of Montreal, Montreal, Canada

³Department of Computer Science and Operations Research, University of Montreal, Canada
lustigo@plk.mff.cuni.cz, aude.dufresne@umontreal.ca,
François.Courtemanche@umontreal.ca

**Abstract.** We present the great potential of real time data, obtained from 1/
mouse and keyboard events' monitoring, 2/ physiological measurements of the
individualized reaction on the learning process of the learning subject in virtual
learning environments. We emphasized simple, non invasive, easily available
methods like eyes tracking, blink rate and blink speed measurements,
electrodermal activities measurements, and heart and/or respiration rate. Those
methods have big potential to reflect decreasing attention, increasing visual or
cognitive information load, task difficulty, tension, stress and fatigue. We
compared the 'real time' data with records obtained from screen captivate SW,
video and audio records, records of external observers and learners (volunteers)
interviews. We highlight the advantages and constraints of different data
acquisition approaches, as well as constraints, done by hardware and software
limits, and discuss the future potential for automated learners' feedback within
VLE.

**Keywords:** virtual learning environment, automated feedback, learning, physiological measurements, data mining

# 1 Introduction

Within the progress of information technologies for education, and the progress of physiological measuring tools, the chance to reflect real and immediate learning and emotional needs of a subject, who learns, has emerged in an inconceivable way.
Simple, noninvasive, low cost measurements of eyes blinks, galvanic skin response (GSR) or heart rate are easily available and with the progress and miniaturization in technologies, transferable to the ‘out of laboratory’ conditions – into the real learning environments – home, classroom, internet coffees, etc.

In the following, we present selected partial experiments (including examples of obtained data sets and their description), which reflect various cognitive brain activities and also emotional stage of the learner, and discuss their advantages and constraints with the goal to reveal their potential for real time individualized reaction
on the learning process of the subject during computer based learning within virtual
learning environment.

## 2 Mouse and Keyboard Events Monitoring and Reflection

The keyboard and mouse activity monitoring is rarely used for educational and
learning purposes in virtual learning environments, although it can bring a full range
of new information, highlight some parts of the learning process and help to improve
virtual learning environment itself. For our purposes we did not used monitoring
tools downloadable from the Internet. We incorporated ‘spying’ software onto Java
applets/physlets, which are traditionally used for teaching physics at our secondary
schools. For additional monitoring and reflection we used also ‘real time’ screen
captivating SW, video and audio records and records of external observers. The time
synchronization (soft real time) of all monitoring tools and methods (mentioned above
and below) is one of the biggest problems, we still solve.

## 3 Eye Tracking and Blinking Patterns

Among many different data, obtained from computer connected eye tracking systems,
the changes in spontaneous blink rate seem to be the most easily detected significant
factor for cognitive and visual information load. Blink rate (BR) is low when
information memory is operating. Blinking is suspended during certain cognitive
activities to avoid disrupting these processes [1], [2].

Wong, Wan and Kaye [2] during their research among eight surgeons found the
significant reduction in the average blink rate between two conditions (casual
conversation and operating, using the microscope -16.69/min and 4.75/min, p= 0.0002
paired t test). We verified their results on a group of eight students (age:19-28).
Surgeons, operating, and students trying to read the values from the onscreen graph
(Fig.: 1), both decreased the average blink rate in a significant way (see table 1).

<table><caption>**Table 1.** Average blink rates (No/min) for students (age 19-22) during periods of casual conversation and while focused on solving physics problem presented as Java applets.</caption><thead><tr><th>Student</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th><th>Mean</th></tr></thead><tbody><tr><td>Casual conversation</td><td>11,8</td><td>17,7</td><td>7,2</td><td>17,6</td><td>13,8</td><td>22,5</td><td>11,2</td><td>13,8</td><td>14,45</td></tr><tr><td>Solving problem</td><td>0,9</td><td>6,4</td><td>0,5</td><td>6,4</td><td>4,7</td><td>7,6</td><td>1,3</td><td>3,8</td><td>3,95</td></tr></tbody></table>

Blinking rate personal pattern is relatively stable in time (Barbato et al. [3])
According some researchers (e.g. [4]) blink rate and especially blink speed reflect
also the level of fatigue. Blinking rate is easy to detect comparing to other eyes
features detection (saccades, fixations, gaze position, etc).
BR measuring also brings some constraints. According Patel at al. [5] observations, for example, visual tasks requiring concentration result in a similar decrease in average blink rate; we obtained from our experiments on cognitive information load. Other researchers [6] also hesitate about BR results, measured under different baseline conditions.

## 4 Galvanic Skin Response

GSR - galvanic skin response is one of the several electrodermal responses. GSR consists of two components: tonic and phasic. The tonic component is a low frequency baseline conductivity level, which can oscillate over the course of days. Each person has a different tonic conductivity, generally in-between 2 and 20 µS. The phasic component rides on top of the tonic component, is of higher frequency, and generally increases when a person is aroused. Tonic skin response rises in anticipation of performing a variety of tasks and during the performance of these tasks (mental arithmetic, attention tasks, discussing social issues).

The low frequency baseline conductivity level (tonic component), which can oscillate over the course of days and generally slowly increases with the level of the person's arousal, is characterized by large individual differences. See Fig. 2a

<figure><img src="image_2.png" /><figcaption>**Fig. 2.** a/The individual differences in both phasic and tonic components of the skin conductance during the exposure to the same web side content. b/ Problems with automated data interpretation, person badly connected, leads to misinterpretation (scaled 0-1 µS). c/ Typical response, illustrating the time parameters for rise time and half recovery time. (scaled 7.5-8,1 µS). Data obtained by authors.</figcaption></figure>

Real time data processing and interpretation of GSR signal is not simple. Phasic component is usually evaluated on the base of amplitude and latent period. For the evaluation of the relative amplitude of the phasic reaction in percent relative to the tonic level is often used. The signals of both, the tonic and phasic component, are not periodic; their spectra are continuous and overlap. For phasic component data interpretation signal processing methods are rarely used, although some methods, e.g. FFT, seem to bring new information [7]. Usually the amplitude in µS, latitude, rise time and half recovery time (typical values in the same order are 3s, 1-3s, 2-10s), are
measured [7]. For illustration see Fig. 3c. Typical problem with computer based real time data interpretation is illustrated on Fig. 2a.

# 5 Heart Rate, EEG, Respiration Rate

While GSR changes are mostly connected with emotional arousal and the level of person's involvement and BR is highly influenced by the level of fatigue and visual acuity, the heart rate and EEG measurements undoubtedly reflect mainly mental activities.

Evidence, since the beginning of 20th century, indicates that alpha power is inversely related to mental. Amplitude decrease in the alpha rhythm, called "alpha blocking," or "alpha desynchronization" has been reported for several cognitive tasks such as mental arithmetic, tasks taken from IQ tests, and creative problems (e.g. [8], [9], [10], [11]). Generally the heart measurements and especially EEG were provided mainly in laboratory conditions. Recently, EEG and HR together with respiration rate measurements improved a lot, and according our experiments seem to have a great potential for the "out of laboratory" large scale use.

# 6 References

1. Holland, M. K., Tarlow, G.: Blinking and thinking. Psychological Reports 41, 403-406 (1975)

2. Wong, K.K., Wan, W.Y., Kaye, S. B.: Blinking and operating: cognition versus vision. British Journal of Ophthalmology. vol. 86(4), pp. 471-479 (2002)

3. Barbato, G., Ficca, G., Muscettola, G., Fichele, M., Beatrice, M., Rinaldi, F.: Diurnal variation in spontaneous eye-blink rate, Psychiatry Research 93, 145-151 (2000)

4. Bittner, R., Smrcka, P., Pavelka, M., Vysoky, P., Pousek, L.: Fatigue Indicators of Drowsy Drivers Based on Analysis of Physiological Signals. Springer. Berlin / Heidelberg. (Print) (Online). vol. 2199/2001. 62-68 (2001)

5. Patel S., Henderson R., Bradley L.: Effect of visual display unit use on blink rate and tear stability. Optom Vis Sci 68. 888-892 (1991)

6. Cho P, Sheng C, Chan C, et al. Baseline blink rates and the effect of visual task difficulty and position of gaze. Current Eye Research 20, 64-70, (2000)

7. Ishchenko, A. N., Shev'ev, P. P.: Automated complex for multiparameter analysis of the galvanic skin response signal. Biomedical Engineering. Springer, New York. 0006-3398 (Print) 1573-8256 (Online). vol. 23, Number 3. May. (1989)

8. Donchin, E., Kutas, M., & McCarthy, G.: Electro cortical indices of hemispheric specialization. In: Lateralization in the nervous system, pp. 339-384. Academic Press. New York (1977)

9. Lustigova, Z.: New e-learning environments for teaching and learning science. In: 20th IFIP World Computer Congress on Learning to live in the knowledge society. pp. 363-364. Springer. Berlin (2008)

10. Martindale, C. Biological bases of creativity. In: Handbook of creativity. pp. 137-152. Cambridge Univ. Press, New York (1999)

11. Nunez, P. L.: Mind, brain, and electroencephalography. In P. L. Nunez (Ed.), Neocortical dynamics and human EEG rhythms, pp. 133-194. Oxford Univ. Press, New York (1995)