Real-time heartbeat interval estimation from face video by multiple filters ensemble
Ryotaro SHIMA, SooIn KANG, and Taketoshi MORI
Graduate School of Advanced Engineering, Tokyo University of Science
Introduction / Background of the Study
The demand for telemedicine and remote health management
for the elderly is increasing. There are high expectations forthe use of small cameras as a method of casually monitoring
health in daily life. Simple home health management methods
include taking body temperature, observing the swelling andredness of the throat, and measuring the respiratory rate. Morerecently, measuring weight and body fat percentage,
measuring blood pressure upon waking up, and measuring
blood oxygen saturation have been adopted. Among these,daily heart rate measurements are becoming particularly
important. Continuous heartbeat monitoring may lead to early
detection and prediction of cardiac problems and other relateddiseases.
Common heart rate monitors or electrocardiographs areprimarily contact methods for the measurement. In
electrocardiographs, electrodes are attached to multiple points
on the body. Most of heart rate monitors are designed as
wristwatch-type or other body-mounted devices. And recently,
heart rate can be measured non-contact and non-invasively byusing a webcam. Various systems have been developed toextract the heart rate by capturing video images of the facewith a smartphone or webcam [1-4]. If it becomes feasible tomeasure heart rate using these, the technique could be applied
to various fields such as health management, stressmanagement, sports training, and medical diagnosis. In
particular, webcams, which are low-cost and widely used atpresent, could be utilized in health management for theelderly and in telemedicine services.
We would like to realize a system that continuously
measures changes in heartbeat intervals over a short period oftime in real time. In previous studies, the heart rate per minuteis usually displayed by Fourier transform based on the green
component from videos lasting several tens of seconds [5].
Difficulties are that the face video contains various noises
such as body movements and lighting fluctuations. Moreover,the heart rate must be calculated at any time based on thepeak-to-peak time of the heartbeat rather than the averageheartbeat over several tens of second to be utilized in various
applications. Additionally, the system should support a wide
bandwidth of heartbeat periods from heavy intense periods tocalm periods to accommodate a wide range of heart rate cycle
bandwidths from fast to slow heartbeats.
In this study, based on the assumption that the time it takes
to recover the resting heart rate after exercise reflects physicalfitness and physical condition of the person, we aimed todevelop a system that can measure exercise function based on
the recovery of the heart rate after exercise by obtaining theheart rate every second in real time, non-invasively andwithout restraints, from facial video obtained with a webcamera. The objective of this project is to realize a system thatcan measure exercise function based on the recovery of heart
rate after exercise.

Fundamentals and Challenges
Measurement from Facial VideoofHeart

Rate

Heart rate measurement by rPPG (remote photoplethysmography), which is a low-cost and non-invasive
method using a camera, is expected to be used in various
fields. However, this method faces challenges due tosensitivity to ambient lighting and subject's movement.
Moreover, it is difficult to imagine a situation in which thesubject remains still for a long period of time, and the subject'smovements may make it impossible to continue capturing thesame skin area, resulting in discontinuous data for a shortperiod of time. Therefore, it is necessary to obtain the heart
rate from data of as short a time as possible. In particular, toobtain the heart rate with a small delay in real time, it isnecessary to extract the peak of the oscillation and calculate
the number of beats backward from the peak interval, rather
than accumulating data for several tens of seconds andestimating the number of beats by filtering in the frequency
domain [6]. In the method of obtaining the heart rate from thepeak interval, a minimum of two peaks is all that is required.
Thus, it is thought that the heart rate can be obtained fromdata of a short duration of at least two seconds. When the heart
rate signal is extracted from a certain skin area on the face, itis necessary to keep tracking the area in order to capture thetemporal changes in skin color. One approach is to detectfacial landmarks such as the nose, eyes, mouth, and facial
contours for tracking the same skin region consistently. While
this method can reduce noise from body movements, it also
has the problem of introducing noise due to differences in thedetection position of landmarks in each frame, which must bedealt with in the same way as noise due to lighting
fluctuations and subject movement. High heart rate data issusceptible to noise due to issues such as the short period of asingle wave, and facial motion due to breathing is alsosignificant. Therefore, the simple way is that a narrow
bandpass filter must be applied to extract the peak of the heartrate. However, a narrow bandpass filter, as in contradiction,cannot cover a wide range of heartbeats, from low to highheartbeats.
System and Implementation of the Proposed Heart Rate
Estimation Method
In this study, we used small web camera (LogitechStreamcam) and obtained facial landmark detection toidentify the facial regions on both cheeks from which theheartbeat signal is easily extracted in a 720p resolution facialvideo at 30 fps, and extracted time-varying data of the average
pixel value of green pixels from which the heartbeat signal
tends to appear. In contrast, a moving average with a windowof 4 frames and a wide bandpass filter from 0.8 Hz to 3.5 Hz
were used to remove noise, and 13 narrow bandpass filterswith different application ranges were used to extract
heartbeats in various heart rate bands from low heart rate tohigh heart rate to facilitate extraction of peaks in each heart
rate band. The bandpass filters ranged from 0Hz to 1Hz,

0.25Hz to 1.25Hz, ..., 2.75Hz to 3.75Hz, and 3Hz to 4Hz,employing 13 bandpass filters shifted by 0.25Hz increments
for a 1Hz range. The method was implemented to calculate 13
heart rates from the past 5 seconds of data every second byfinding the peaks in the data after applying these filters anddetermining the average peak interval after removing outliers
from that peak interval (see the three figures below). A
histogram of heart rates in increments of 15 beats per minuteis created based on the maximum past 30 seconds of heart ratedata for each filter, the largest cluster is estimated as thecurrent heart rate band by K-means clustering, and the heart
rate when the bandpass filter that can extract that heart rateband is used as the current heart rate (Fig. 1).

means. The histogram of the area where the heart rate band
was incorrectly identified showed that small clusters of noise
were clustered together, creating clusters larger than thecurrent heart rate cluster. We considered that this was causedby the width of the histogram bins being too wide, so wechanged the bin width from 15 beats per minute to 10 beats
per minute and conducted an experiment with the same data.
The adjustment improved the correlation coefficient from0.753 to 0.831.
180

heart rate[BPM]1601401201008060
0102030405060time [s]subjectX_exerciseA_1_ECG_HR

subjectX_exerciseA_1_face_HR

200
190

180

170
160

160

150

heart rate [BPM]

heart rate [BPM]

180

140
130
120
110
100
0100

200

300

400

500600time [s]Fig 1. Screen shot of heart rate measurement (above) andoutput of beats every second (lower left), and heart rate
every second after applying the filter (lower right)

14012010080600102030405060time [s]subjectY_exerciseA_1_ECG_HR

Methods and Results of the Verification ExperimentsubjectY_exerciseA_1_face_HRFig 2. Comparison of real-time heart rate measurementsof subjects X (above) and Y (below)

To verify whether the realization system can accurately
measure real-time heart rate, the heart rate was compared with
the ECG heart rate measured by an electrocardiograph(Checkme ECG ADV, SAN-EI MEDISYS). For video
recording, each subject was recorded in three sets: one at rest
and two after exercise. Within a 10-minute time period, weperformed five consecutive 60-second ECG and video
recording measurements, defining this sequence as one set. Todetermine whether the heart rate changes after exercise could
be captured, and whether the heart rate in various heart ratezones could be captured, we acquired the resting and postexercise heart rates of two subjects. For the validation
experiment, we compared the ECG heart rate displayed on theelectrocardiograph during 60-second video with the heart rate
calculated by our system in the face video for each second.
We verified whether the system can accurately measure fortwo subjects with and without individual differences. Figure
2 shows an example of time variation after exercise forsubjects X and Y. Figure 3 shows a scatter plot of the ECG
heart rate and the method used in this study. The correlation
coefficient between the facial animation and ECG heart rate
was 0.753. The correlation coefficient of the post-exercise
data alone was 0.93, and that of the resting data alone was0.041. Outliers were more frequent at low heart rates of lessthan 90 beats per minute, and fewer at high heart rates of 130
beats per minute or more. We considered that the instability
at low heartbeats was due to instability in the identification of
the rough frequency region where heartbeats existed by K-facial animation heart rate [BPM]

170

150

130
y = 1.0252x
R² = 0.9748
110907050507090110

130150170

ECG heart rate [BPM]

Fig 3. Scatter plots of ECG heart rate and facial animation
Our developed system successfully followed the change inthe heart rate of each subject after exercise as it approached
the resting state (see figure below). Subjects X and Y showed
differences in heart rate recovery time. However, the exercise
load of the two subjects was not controlled, and it could not
be verified whether there was a difference in physicalrecovery ability.

2180

Conclusion

160

heart rate [BPM]140

In this study, we achieved real-time peak-to-peak heartbeat
interval estimation by combining three methods: (1) a method
to limit the heartbeat information acquisition area by
extracting feature points from a face image and estimating theface shape, (2) a method to narrow the heartbeat intervalrange by identifying the cycle bands as a broad mass byclustering, and (3) a method to calculate the heartbeat interval
sequentially by tracing the heartbeat change. In an experimentusing a 30fps 720p web camera, a high correlation of 0.753
was obtained in comparison with a portableelectrocardiograph. The correlation coefficient increased to0.831 by changing the width of histogram bins from 15 beats
per minute increments to 10 beats per minute increments.
Increased facial movement and changes in ambient light will
result in more noise mixed into the heart rate signal; thus, ahigher frame rate, more bit depth, and better resolution ofcamera image are expected to provide more accurate andfrequent heart rate measurements.

120
100
80
60
40
20
000:00

01:00

02:00

03:00

04:00

05:00

06:00

07:00

08:00

09:00

time [MM:SS]
subjectX_exerciseA_face_HR

subjectX_exerciseA_ECG_HR

180160heart rate [BPM]

140
120
100
80
60
40
20
000:00

01:00

02:00

03:00

04:00

05:00

06:00

07:00

08:00

09:00

time [MM:SS]
subjectY_exerciseA_face_HR

subjectY_exerciseA_ECG_HR

ReferenceFig 4. Tracking of changes in heart rate recovery time forsubjects X (above) and Y (below)[1] Sebastian Zaunseder, Alexander Trumpp, Daniel Wedekind,
Hagen Malberg, "Cardiovascular assessment by imaging
photoplethysmography – a review", Biomed. Eng.-Biomed. Tech.,vol. 63, no. 5, pp. 617-634, 2018.
[2] M.A. Hassan, A.S. Malik, D. Fofi, N. Saada, B. Karasfi, Y.S. Ali,
F. Meriaudeau, "Heart rate estimation using facial video: A review",
Biomedical Signal Processing and Control, vol. 38, pp. 346-360,
2017.
[3] Xun Chen, Juan Cheng, Rencheng Song, Yu Liu, Rabab Ward, Z.
Jane Wang, "Video-Based Heart Rate Measurement: RecentAdvances and Future Prospects", IEEE Transactions on
Instrumentation and Measurement, vol. 68, no. 10, pp. 3600-3615,
2019.
[4] Serge Bobbia, Richard Macwan, Yannick Benezeth, Alamin
Mansouri, Julien Dubois, "Unsupervised skin tissue segmentation
for remote photoplethysmography," Pattern Recognition Letters, vol.
124, pp. 82-90, 2019,
UBFC-RPPG Dataset(https://sites.google.com/view/ybenezeth/ubfcrppg).
[5] Wim Verkruysse, Lars O Svaasand, J Stuart Nelson, "Remoteplethysmographic imaging using ambient light", Opt Express, vol.
16, no. 26, pp. 21434-21445, 2008.
[6] Yuichiro Maki, Yusuke Monno, Kazunori Yoshizaki, MasayukiTanaka, Masatoshi Okutomi, "Inter-Beat Interval Estimation fromFacial Video Based on Reliability of BVP Signals", 2019

Discussion
The system developed in this study still has an issue whereeven after changing the bin width of the histogram, theestimation accuracy for low heart rates remains low, resulting
in the calculation of values that are clearly higher than thecorrect heart rate. To solve this issue, one idea is to firstroughly distinguish between low and high heart rates byapplying only a wide range of bandpass filters. Additionally,
there is a possibility that the width of the bandpass filter was
set to increase the estimation accuracy of high heart rates, so
there is a chance for improvement by reviewing and tuning
the system parameters.
This study was conducted based on the hypothesis thatexercise function can be measured from the recovery time ofheart rate after exercise. By using this system, it was possible
to track the recovery of heart rate after exercise and confirm
differences in heart rate recovery time. However, it has notbeen verified and is unclear whether it is actually possible toexamine exercise function from the time it takes for the heart
rate to recover and settle down to a certain extent after
exercise. Therefore, it remains a future task to investigate
whether there is a relationship between the heart rate recovery
time after exercise and exercise function by aligning theexercise load of each subject.

3