立石科学技術振興財団

助成研究成果集(第27号) 2018

画像からの心拍数計測と表情認識を用いた感情状況の変動推定
Vision Based Emotion Estimation Using Heart Rate and Facial Expression
2161027

研究代表者

埼玉大学

大学院理工学研究科

助教

LAM

Antonyenjoy it because they were being polite. Thus it[Purpose of Research］

is important to use some other means of sensing
The ability to read human emotion has manyemotion to supplement facial expressions.

applications in human-computer interaction

Fortunately, psychophysiology [1] indicatesand affective computing. Reading facial expres-emotions can also be recognized by measuring

sions is a common approach in emotion expres-physiological responses like cardiac activity.

sion recognition systems. However, there are

Physiological responses due to emotions aretimes when people do not show obvious facial

also more difficult to hide. However, mostexpressions (Fig. 1). There are also times when

proposed systems rely on specialized sensors

people may try to hide facial expressions. For

that must be attached to the body. This limits

example, if we asked people to participate in a

the range of applications possible.

pilot viewing of a new TV show, they mayIn our work, we propose using video-based

indicate it was enjoyable even if they did not

photoplethysmography to read cardiac activity
from RGB videos of the human face. Due totiny color changes in the skin, reading cardiac
activity from RGB is possible and we show its
effectiveness in emotion detection/recognition
on video content.
From our findings, it will then be possible touse cardiac-based emotion recognition to supplement facial expression recognition. A major
upcoming application of our work is in detecting
the emotional state of elderly patients in carefacilities using our sensing system. We are
currently investingating this application incooperation with an elderly care facility.

Fig. 1

Human subject watching video clips. Facial
expressions are similar. We recognize emotion byphotoplethysmograms (PPGs).

― 1 ―立石科学技術振興財団

lation to track triangular regions on the face[Contents and Results］(instead of square patches) to address poseIn the early phases of our work, we took ourpast “heart rate from video” algorithm [2] as achanges. We also used the RGB channel
weighting scheme proposed in [3].

starting point and explored ways to improve

To confirm the feasibility of estimating HR

robustness in more realistic settings such asfrom YouTube, we captured 18 videos withmotion and poor lighting (1, 2, 6). (Note thathighly varied motion and uploaded them tocitations with parentheses refer to our “Publi-YouTube. We then downloaded our videos fromcations and Achievements”.)

YouTube and estimated HR. We found that theLater, we built a dataset of human subjects

uploaded YouTube videos had a mean absolute(40 university students) watching horror anderror (MAE) of 9.9 BPM. The original videos

comedy video clips and measured their heart

had a 6.9 BPM MAE but it shows HRrates (HR) from videos of their faces. (3) Inestimation from YouTube should be possible.

this study, we verified that the emotional

We then downloaded videos from sevenstimulation from videos cause a statistically

vloggers that watched the Christmas Horror

significant increase in average HR compared toMovie, “Krampus” (Fig. 3).

the subjectsʼ average resting HR. We also

From these videos, we estimated the average

confirmed that our video PPG algorithm could

HR (over all the vloggers) at 30-second inter-accurately read HR compared against a FitBit

vals and observed the changes in average HR.

sensor (correlation 0.9). The possibility of using

Fig. 4, shows, in the scariest parts of the trailer,

video PPG to detect emotional changes in

the average HR had a significant increase (withmultiple people using a single camera was also

a decrease in standard deviation). This sug-investigated. (4)gests we can read accurate HR with detected

Our next step was to explore reading cardiac
activity from real world videos and see if it

emotional changes consistent with our pastfindings on our dataset from (3).

would be possible to detect emotional states.

Despite the effectiveness of using HR toSpecifically, we decided to use YouTube

detect emotional changes, there is the limitation“reaction vlogs” as a testbed of real-worldthat HR does not give enough information tovideos.

“Reaction vlogs” are videos wherevloggers record themselves watching things
such as movie trailers. We can see their facial
reactions and also try to estimate their cardiacactivity using our video PPG algorithm.
We did a preliminary study in (5) onFig. 2

YouTube video analysis. To deal with theBasic flow of the video PPG algorithm forestimating cardiac activity.

complex motions that could occur in real-world
videos, we also introduced some major improvements to our algorithm (Fig. 2). Thisalgorithm is similar to our past work [2].
However, we introduced Delaunay Triangu― 2 ―Fig. 3Sample “YouTube Reaction” Videos

Tateisi Science and Technology Foundation

Table 1 Leave One Out Cross-Validation results using linearSVMs in all cases. PCA denotes PCA preprocessingof the data.
Feature TypeLOOCV Acc.

Video PPG
Video PPG (PCA)

65.4 %
67.3 %

Facial Action Units

76.9 %

Facial Action Units (PCA)

78.8 %

not ask the subjects to hide their facial
expressions. Our method is still able to readFig. 4 Average HR (BPM) of the vloggers from watching
the movie trailer. Standard deviation bars are shown.

emotion relatively well without relying on facial
expressions, which can be easily hidden.

classify the type of emotion a person experi[Future Work］

ences. For example, we may want to know if
someone is experiencing fear or joy. Thus in(7), we decided to estimate the PPG signal

We are working with a care facility to(cardiac pulse signal and not just HR, see Fig.

explore applications in elderly care monitoring.

1) and then attempt to classify emotions.

We will also work towards improving video

In (7), we were able to estimate the PPG

PPG-based emotion sensing and how it can besignal (not just the HR). Our algorithm (Fig. 2)

effectively combined with facial expression

provides multiple independent hypotheses ofrecognition.

the PPG signal. We found that by taking thenormalized average PPG signals (with thesame dominant HR), we could get a single
robust estimate of the PPG signal. The PPGsignals in Fig. 1 are examples of actual
estimates from video.

[Publications and Achievements］
1)

A. Lam, Y. Kuno, “Towards Estimating Heart
Rates from Video Under Low Light”, In 12 th Intl.
Symp. on Visual Computing, Springer, 2016.

2)

We then applied this approach to estimate
the PPG signals on our dataset of university
student reactions to watching horror andcomedy video clips. We then tested using SVMs
to classify the PPG signals into either “horror”
or “comedy”. (Being able to classify PPG
signals by the type of video the subject watchedwould be an indication of effective emotion
classification.) Table 1 shows classification
results using PPG signals estimated from ouralgorithm and compares to using the average
instensities of 17 facial action units. Facial
expressions gave better results than ourPPG-based method but this is because we did― 3 ―大 津 耕 陽，倉 橋 知 己，K. Das，福 田 悠 人，A.
Lam，小林貴訓，久野義徳，“環境変化に頑健なビデオ映像による心拍数計測手法”，In 23 rd Symp. onSensing via Image Information, 2017. (最優秀学術論文)
3)

K. Das, S. Ali, K. Otsu, H. Fukuda, A. Lam, Y.

Kobayashi, Y. Kuno, “Detecting Inner Emotions
from Video Based Heart Rate Sensing”, In Intl. Conf.
on Intelligent Computing, Springer, 2017.
4)

K. Das, K. Otsu, A. Lam, Y. Kobayashi, Y. Kuno,“Towards Detectng the Inner Emotions of MultiplePeople”, 第 99 回パターン計測部会研究会，2017.
5)

A. Lam, K. Otsu, K. Das, Y. Kuno, “Towards

Taking Pulses Over YouTube to Determine Interest
in Video Content.” In Intl. Workshop on Frontiers ofComputer Vision, 2018.
6)

K. Otsu, K. Das, H. Fukuda, A. Lam, Y. Kobayashi,

Y. Kuno, “映像解析に基づく頑健・高速な心拍数計測手法”, In 24th Symp. on Sensing via Image Infor-

立石科学技術振興財団

mation, 2018.
7)

physiology”, Psychophysiology. Jul, 1 : 90-1, 1964.

K. Das, A. Lam, H. Fukuda, Y. Kobayashi, Y. Kuno,[2]

A. Lam, Y. Kuno, “Robust Heart Rate Measure-“Classification of Emotions from Video Basedment from Video Using Select Random Patches”, In

Cardiac Pulse Estimation”, In Intl. Conf. on Intelli-Intl. Conf. on Computer Vision, 2015.

gent Computing, Springer, 2018.

[3]

G. de Haan and V. Jeanne, “Robust Pulse Ratefrom Chrominance-Based rPPG”, IEEE Trans. on
Biomedical Engineering, vol. 60, no. 10, pp. 2878-[References］[1]

2886, Oct. 2013.

J. A. Stern, “Toward a Definition of Psycho-― 4 ―