# Analysis of Facial Expressions for the Estimation of Concentration on Online Lectures

Renjun Miao, Haruka Kato, Yasuhiro Hatori, Yoshiyuki Sato, Satoshi Shioiri

<figure><img src="image_1.png" /></figure>

## To cite this version:

Renjun Miao, Haruka Kato, Yasuhiro Hatori, Yoshiyuki Sato, Satoshi Shioiri. Analysis of Facial Expressions for the Estimation of Concentration on Online Lectures. IFIP World Conference on Computers in Education (WCCE), Aug 2022, Hiroshima, Japan. pp.343-348, 10.1007/978-3-031-43393-1_31. hal-05533288

HAL Id: hal-05533288

https://inria.hal.science/hal-05533288v1

Submitted on 2 Mar 2026

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_4.png" /></figure>

# Analysis of Facial Expressions for the Estimation of Concentration on Online Lectures

Renjun Miao(✉) ID, Haruka Kato ID, Yasuhiro Hatori ID, Yoshiyuki Sato ID, and Satoshi Shioiri ID

Tohoku University, RIEC Main Building, 2-chōme-1-1 Katahira, Aoba Ward, Sendai, Miyagi, Japan

{miao.renjun.s1,haruka.kato.p8,yhatori,yoshiyuki.sato.e4, satoshi.shioiri.b5}@dc.tohoku.ac.jp

**Abstract.** The present study aimed to develop a method to estimate the state of attention from facial expressions while students are in online lectures. We conducted an experiment to measure the level of attention based on reaction time to detect an auditory target, which was the disappearance of noise sound, while watching lecture videos, assuming that reaction time for the detection of contents-irrelevant noise is longer when learners are focusing attention more to the contents of the videos. We sought facial features that are useful for predicting the reaction time and found that reaction time can be estimated in some amount from facial features. This result indicates that facial expressions are useful for predicting attention state, or concentration level while attending lectures.

**Keywords:** Attention · Facial Features · Online Lecture

## 1 Introduction

In order to improve education quality, it is critical to estimate the level of learners' concentration on studies. For online learning, a web camera can be used easily for recording learners' faces and the facial expressions can be used for estimation of concentration levels. There are several attempts to evaluate mental states with facial expressions. For example, there are reports that attempted to estimate image preference from facial expressions [1-3], which suggest that facial expressions are useful features for estimating subjective judgments of image preference. Similarly, Thomas and Jayagopi recorded images of students' faces in a classroom while they were studying with video material on a screen and estimated the level of engagement from students' facial expressions [4]. Their engagement prediction showed a certain level of accuracy suggesting that facial expressions are useful for estimating the level of engagement or concentration as well. However, previous studies, including Thomas and Jayagopi, were based on subjective judgments. It is crucial to develop a method to estimate the level of concentration objectively. In this study, we focused on estimating attention states using a typical psychophysical measurement, that is, reaction time to detect a target. We designed an
experiment, in which participants were asked to detect an auditory target while watch-
ing a lecture video. Understanding the lecture was the primary task and participants
answered quizzes after watching the video. Detection of the target was the secondary
task and reaction time was used as an objective measure of attention levels when watch-
ing the lecture videos, assuming that reaction time to a target that is not related to the
primary task is longer when participants are focusing on the videos. Images of partici-
pants' faces were recorded while they were watching the videos and facial expressions
were analyzed after the experiment. The purpose of the study is to estimate the reaction
time from the facial expressions to develop the method to estimate concentration level
from images of learners' faces.

2 Experiment

The aim of the experiment was to obtain reaction times to an auditory target as a measure of attention level. Simple reaction time to the appearance of light or sound is used to estimate how much attention is paid to the target (shorter reaction time if more attention is paid to a target). Although this assumption could be too much simplified and the other factors should also be considered (see Discussion), we started with this for the first attempt. We used the disappearance of a continuous noise sound as the target. The reason we used disappearance instead of presentation was to avoid the influence of bottom-up attention, which could reduce the effect of attention on the lectures because salient stimulus attract attention independently of other factors such as concentration on the lectures.

Nine participants took part in the experiment and their average age was 22.8. They had normal or corrected-to-normal vision and normal audition. The task of the experiment was to detect changes in white noise (whose frequency spectrum is uniform) sound while watching online learning material. The target was the disappearance of the white noise. The learning materials were the introductory PHP (PHP is a computer language) course with nine lectures on YouTube [5], and we describe watching one lecture as one session. Participants watched videos on a computer display (MacBook Pro retina 15inch late 2013) with headphone (Sony dynamic stereo headphones MDR-7506) in an experimental booth, with no light source except for the display. Average loudness of lecturer's voice was 70 db and that of white noise was 0.66 db.

Participants were instructed to watch each video lecture to learn PHP and respond to quizzes after watching each lecture. They were also instructed to press a key when they noticed the auditory target, the sudden disappearance of the white noise. This was a secondary task to estimate the attention state at the time of target presentation. The participants were informed that understanding the lecture was the main task of the experiment. The white noise disappeared after the period randomly selected between 25 and 35 s. The white noise started again right after the key press for the detection and started after a pause of 10 s when there was no key press. There were nine sessions of watching one lecture, and one lecture time was between 10 and 20 min. At the end of each lecture, a set of quizzes were provided to evaluate the learning effect (Fig. 1).

There were two control sessions, where the participants were not asked to watch the
video displayed, but to focus on white noise. The control session was to evaluate the
<figure><img src="image_5.png" /><figcaption>Fig. 1. Experimental design.</figcaption></figure>

detection task per se in the condition without paying attention to the lecture. In the control session, the same video lectures were used so that the participants knew the content and had little or no reason to be attracted to the content. Each participant participated in nine lecture sessions and two control sessions, and the total time of watching videos was about 134 min. The experiment was conducted in two days. The first control session was conducted at the 6th session with the video of the first lecture repeated and the second control session was conducted at the 11th session with the video of 6th lecture. Five lecture sessions and the first control session were performed on the first day and the remaining four lecture sessions and the second control session were performed on the second day.

## 3 Facial Feature Analysis

Videos were recorded of participants' faces during the lectures and facial features were analyzed after the experiment. We analyzed video data frame by frame, using OpenFace [6] to analyze the facial feature as action units (AU). AUs of participants' faces from 3 s to 0 s before the target presentation (disappearance of white noise) were analyzed to search for facial expressions that predict reaction times, with which, we assume, the level of attention can be estimated at around the time of target presentation.

The Facial Action Coding System (FACS) [7] is a system that displays a series of facial muscle movements corresponding to the displayed emotion. An Action Unit (AU) code is assigned to each facial movement. For example, AU1 indicates the raising of the eyebrows; AU4 indicates the lowering of the eyebrows.

To predict reaction time from facial expression, we used a machine learning method, Light GBM [8], to model the relationship between AUs and reaction time.

Light GBM is a gradient boosting model, which is considered to be fast with relatively accurate performance. This choice of Light GBM followed a previous study which used several other methods and found similar results [1–3]. For the model evaluation, we used a five-fold cross-validation method. All the data were divided into 5 groups and 4 of them were used for training and the remaining one was used for the test. In the procedure, each group was tested once and used for training four times. The average of the five test scores was used as the final score. The prediction performance was evaluated by root mean square error (RMSE) from the prediction and the coefficient of correlation between data and prediction.
The features of the facial expressions were extracted throughout each session from the recorded video, using an open-source facial expression analysis tool, OpenFace, which automatically recognizes faces and analyzes facial landmarks, head orientation, and gazes. Two types of AU outputs are available: 0–5 intensity of 17 different AUs (AUr) and a binary value of 18 different AUs (AUc).

We used AU data between 3 s and 0 s before the target presentation (disappearance of white noise) for statistical indexes of AU values of the period, which were the mean, minimum, maximum, standard deviation, 25%, 50% (median) and 75% for AUr and means and standard deviation for AUc. The total number of indexes were 155 (17 AUr × 7 and 18 AUc × 2) (Fig. 2).

<figure><img src="image_6.png" /><figcaption>Fig. 2. Analysis of facial expressions.</figcaption></figure>

# 4 Results

The average reaction time of all participants was 1.63 s on all lecture sessions and the standard deviation was 2.186. Since the reaction time varied among participants, we normalized the reaction time data as Z-scores after taking logarithm, which was to reduce the asymmetry in the distribution. Using the normalized values of AUs, we applied Light GBM for modeling the relationship between reaction time and facial expressions, and tested the model by the five-fold cross-validation method. Data without response within 10 s were excluded. Such target presentations were 1.03 on average.

The left side of Fig. 3 shows the relationship between the reaction time results (horizontal axis) and the prediction from Light GBM (vertical axis). Each point represents each target presentation from all sessions of all participants and different colors indicate different training-test combinations (five colors in the figure). The RMSE of data deviation from the predictions is 0.7, which is smaller than RMSE from the average, that was 1 after normalization. The correlation between data and predictions was statistically significant ($p < 0.001$, $t = 38.2$, $r = 0.716$). These results suggest that facial expressions can be used to predict the participant's reaction time to the target that was irrelevant to the contents of the lectures. We assume that reaction time to the target varies depending on attention states so that we claim that facial expressions contain useful information to estimate the level of attention while learning in lectures. That is, the prediction of reaction time, in turn, predicts the attention state during learning.
A further analysis revealed the contribution level of each AU to the prediction, that is, how much each AU is important for the prediction: Sharpley additive explanations (SHAP). Figure 3b shows the results of SHAP. AU45 (Blink), AU15 (Lip Corner Depressor) and AU7 (Lid Tightener) are the three largest contributors among all AUs. The Discussion provides consideration of why these three indexes are important.

<figure><img src="image_7.png" /><figcaption>**Fig. 3.** (a) Correlation between measured reaction time and prediction of the model. (b) Indexes are arranged according to the level of contribution to the prediction obtained using a method (SHAP). The color, red/blue, indicates positive/negative contribution. (Color figure online)</figcaption></figure>

# 5 Discussion

In this study, we used a machine learning technique to predict the reaction time for detecting an auditory target using facial images of participants who were watching video lectures. From the reaction time, we can infer how attentive a person is: a slower response suggests that the person is highly attentive to the lecture, paying less attention to the target. We identified facial features that contribute the most to the prediction, which are the action units of AU45 (Blink), AU15 (Lip Corner Depressor), and AU7 (Lid Tightener). We speculate the reasons why they are important features to predict reaction time. Blinks are known to be related to task given [9], and our results suggest that attention states could be estimated from blink. AU15 (Lip Corner Depressor) may be related to the condition where a person feels difficulty or confusion to understand the lecture and tries to attend more to the lecture. AU7 (Lid Tightener) may be related to sleepiness because the brow corners are tense when people are sleepy. These speculations may be valid, but further investigation is required with a larger number of participants to clarify the issue, since there are significant individual differences for the AUs of largest contribution. We should be cautious about generalizing the present results.

The assumption of reaction time and attention to the lecture could be too simplified. Longer reaction time to the target can be due to lowering the general arousal level, for example, instead of stronger attention to the lecture. In our control condition, participants were asked to detect the target without paying attention to the video lecture. This reaction time can be considered as reaction time with full attention to the target detection. The average reaction time across all participants was 0.68 s for the two control sessions.
The time is much shorter than 1.63 s, which is average reaction time of lecture sessions. Although this does not deny the possible influence of change in arousal level on the reaction time for lecture sessions, reaction time to the target can be a good measure to estimate attention level to the lectures. We also analyzed the results removing data of sessions where participants appeared to be sleepy. By watching the face videos of all sessions of all participants, the first author judged whether each session might have been influenced by sleepiness or not. The similar prediction performance was obtained for the data to the original results, suggesting that sleepiness or arousal level is not major factors of reaction time variation. Since this was an unofficial attempt, we do not conclude that the arousal level did not influence the present results. Only further investigation can make the issue clearer. Indeed, an analysis of the reaction time change with the progress of the lecture during a session showed a trend of increase in reaction time as the lecture progressed. This may indicate attention changes following the progress of a lecture toward the direction of reducing attention, which may be interpreted by change of arousal level or effect of sleepiness.

In conclusion, we have developed a method to evaluate attention state using facial expressions of learners. This method can be used to improve education quality when learners' face images are available, such as during online lectures.

## References

1. Shioiri, S., Sato, Y., Horaguchi, Y., Muraoka, H., Nihei, M.: Quali-informatics in the society with yotta scale data. In: 2021 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1–4 (2021)

2. Sato, Y., Horaguchi, Y., Vanel, L., Shioiri, S.: Prediction of image preferences from spontaneous facial expressions. Interdiscip. Inf. Sci. 28(1), 45–53 (2022)

3. Horaguchi, Y., Sato, Y., Shioiri, S.: Estimation of preferences to images by facial expression analysis. IEICE Tech. Rep. 120(306), HIP2020-67, 71–76 (2020)

4. Thomas, C., Jayagopi, D.B.: Predicting student engagement in classrooms using facial behavioral cues. In: Proceedings of 1st ACM SIGCHI International Workshop on Multimodal Interaction for Education, pp. 33–40. ACM, New York (2017)

5. https://www.youtube.com/watch?v=uVaOzQLXt0. Accessed 02 Oct 2022

6. Baltrušaitis, T., Robinson, P., Morency, L.P.: OpenFace: an open source facial behavior analysis toolkit. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–10. IEEE, New York (2016)

7. Ekman, P., Friesen, V.W.: The Facial Action Coding System: A Technique for The Measurement of Facial Movement. Consulting Psychologists Press, San Francisco (1978)

8. https://lightgbm.readthedocs.io/. Accessed 02 Oct 2022

9. Doughty, M.J.: Consideration of three types of spontaneous eyeblink activity in normal humans: during reading and video display terminal use, in primary gaze, and while in conversation. Optom. Vis. Sci. 78(10), 712–725 (2001)