SEQuence-rPPG: A Fast BVP Signal Extraction Method From Frame Sequences
Authors: Kegang Wang
Yantao Wei
Mingwen Tong
Jie Gao
ZhongJin Zhao
YuJian Ma
Yi Tian
Venue: Submitted to ICLR 2023 (2023)
Keywords: rPPG
Remote vital sensing
Signal processing

Abstract: Non-contact heart rate estimation has essential implications for the development of affective computing and telemedicine. However, existing deep learning-based methods often endeavor to achieve real-time measurements, so a simple, fast, pre-processing-free approach is needed. Our work consists of two main parts. Firstly, we proposed SEQ-rPPG, which first transforms the RGB frame sequence into the original BVP signal sequence by learning-based linear mapping and then outputs the final BVP signal using 1DCNN-based spectral transform, and time-domain filtering. Secondly, to address the shortcomings of the existing dataset in training the model, a new large-scale dataset was collected for training and testing. Our approach achieved competitive results on the collected large dataset(the best) and public dataset UBFC-rPPG(0.81 MAE with 30s time window, test only). It requires no complex pre-processing, has the fastest speed, can run in real-time on mobile ARM CPUs, and can achieve real-time beat-to-beat performance on desktop CPUs. Benefiting from the high-quality training set, other deep learning-based models reduced errors by at least 53$\%$. We compared the methods with and without the spectral transformation, and the results show that the processing in the time domain is effective.

Under review as a conference paper at ICLR 2023
SEQ-RPPG: A FAST BVP SIGNAL EXTRACTION METHOD FROM FRAME SEQUENCES
Anonymous authors Paper under double-blind review
ABSTRACT
Remote photoplethysmography (rPPG) can be widely used in various kinds of physical, health, and emotional monitoring, such as monitoring the heart rate of drivers, consumers, the elderly, and infants. Several rPPG methods have been proposed in the past few years, but non-contact heart rate estimation in realistic situations is still challenging. It is observed that the existing deep learning- rPPG methods can not achieve real-time performance on low-cost devices. To deal with this problem, a simple, fast, and pre-processing-fie free approach called sequence- based rPPG (SEQ-1PPG) is proposed for non-contact heart rate estima- tion. SEQ-rPPG first transforms the RGB frame sequence into the new signal sequence by learning-based linear mapping and then outputs the final BVP sig- nal using IDCNN-based spectral transform and time-domain filtering. It requires no complex pre-processing, has the fastest speed, can run in real-time on mo- bile ARM CPUs, and can achieve real-time beat-to-beat performance on desktop CPUs. Furthermore, We present well-annotate dataset, focusing on construct- ing large-size and highly synchronized PPG and video. The entire data set will made available to the research community. Benefiting from this high-quality dataset, other deep learning-based models reduced errors. To prove the efficacy the proposed method, the comparison is done with state-of-the-art methods. The experimental results on both self-build and publicly available datasets have demonstrated the effectiveness of the proposed method. We also verified that the processing in the frequency domain is effective.
INTRODUCTION
The Blood volume pulse (BVP) a physiological measurement used to extract physiological signals from the heart. The most used method of BVP signal extraction is photoplethysmogram (PPG). However, PPG requires the subject to wear an optical contact sensor, which can cause discomfort and lead to difficulties, such as monitoring patients in pediatric intensive care units. Non-contact BVP extraction is possible via high-sensitivity cameras and webcams using ambient light as a source of illumination Takano & Ohta 2007 Verkruysse al. 2008). So remote PPG (rPPG) has attracted extensive attention in recent years.
Early rPPG methods mainly focused on temporal modeling. For example, independent component analysis (ICA) Poh et al. (2010a) is used for extracting BVP signals from RGB signals. Further- more, Poh al. (2010b) added the detrending operation improve the robustness of ICA. But the blind source separation techniques in RGB color space show limited success. de Haan & Jeanne (2013) propose a chrominance-basee ral modeling focused on linear optical models and filter-based post-processing, which empirically proved to fast and effective. addition, spatial modeling also an important way to extract BVP Tulyakov et al. (2016); Bobbia et al. (2019). They attempt to select the part of the face with the highest signal-to- toiise ratio from different regions of the face. This way, it can accommodate more head movements, illumination, and shading. However, these handcrafted algorithms with poor accuracy compared to deep learning- based approaches.
method that is robust to luminance. We can find that tempo-
Recently, deep leamning-base based approaches (Chen & McDuff 2018 Niu et al. 2020a b Liu al. 2020 [Sog et al., [022]: Yu et al. 2019 Lu et al., [221] Liu et al.) 2021 Yu et al. 2022 have achieved better accuracy. However, the computation cost of these methods much larger than

---

Under review as a conference paper at ICLR 2023
handcrafted algorithms, making it difficult deploy low-cost devices. The high computation cost comes from two sources. First is the pre pre-processing. In Niu et al. (2020a by: Song et al. (202T) Lu al. (02T), algorithms require fine segmentation of the face, so these algorithms rely on handcrafted features, and the computation cost required for pre-processing is even greater than the model itself. Second, some end-to-end models(Liu et al., [021] Yu et al., 2022) achieve high accuracy, but the models are too large to run on devices without GPUs. Although some models Yu al. 2019; Liu et al., 2020) can run in real-time on mobile CPUs with good accuracy, the computation cost is still too high. rPPG is commonly used in affective computing and telemedicine, where users must juggle other components while running rPPG applications. It is unacceptable to allocate most of the computation resources to real-time rPPG. Therefore, designing rPPG algorithm that only takes up a few computation resources is necessary.
Apart from the algorithm design, the dataset is also vital for obtaining better learning models. Dif- ferent training sets have a huge impact on model performance, the main reason is that most of the datasets are not well synchronized with the video signal and the BVP signal. There are some datasets that are highly synchronized, such as PURE, SCAMPS Strickke et al. 2014 McDuff et al. 2022). However, PURE only has 59 minutes of video, and SCAMPS has not yet shown strong enough gen- eralization as a synthetic dataset. In addition, some works (Yu et al.) 2019: Botina- Moanmaa-Monllllaaa ve et al.] making the model adaptable to non-synchronized datasets, more common practice is to use unsupervised methods to synchronize signals (eg. POS), but is still an open problem design methods robust non-synchronization. Although there are different ways to alleviate the problem of non-synchronized, the scarcity of high-quality datasets still limits the performance of some models.
2022 Comas al., 2022) focused
To deal with the problems mentioned above, an effective BVP extraction method is proposed, and a new dataset constructed. The main contributions of this paper can be summarized follows:
A simple, fast, and pre-pr pre-processing-feee free method is proposed for non-contact BVP extrac- tion. The proposed method, called sequence-basedd rPPG (SEQ-IPPG), uses linear map- ping, spectral transform, and time-domain filtering to extract BVP signals. The proposed method requires no complex prepprocessing and can on mobile ARM CPUs in real- time. Its computation cost and memory usage are much smaller than existing algorithms.
A large, synchronized dataset is built for better model training. In this paper. we developed software that simultaneously obtains the ground truth signal from the blood pulse meter and the RGB signal from the webcam, and all UNIX timestamps are recorded. It is designed for large-scale training, more than 30 hours (3. 3.24M frames) of video available, and larger in duration than any public dataset.
2 RELATED WORK
2.1 END-TO-END PHYSIOLOGICAL SIGNAL MEASUREMENT NETWORK
removing it and are vague in their end-to-end definitions, and they all claim to be end-to-end models. In DeepPhys (Chen & McDuff 2018) and MTTS-CAN (Liu et al., 2020)), the pre-processing algorithms generate differen- tial and average frames, respectively, then enter them into the motion branch and appearance branch of the network. In EfficientPhys (Liu et al. 202I). they use a rre-processing approach similar to MTTS-CAN, yet within the model and determine the normalization parameters by learning. It is hard to say if this is pre- pre-prceessing or part of the model internals. In PhysNet (Yu et al., 2019) and PhysFormer YYu et al. 2022), the model accepts original image input directly without additional pre-processing. MITT-CAAN is an improved version of DeepPhys, which has higher accuracy, and although they both require pre-processing, the pre- pre-processing is simple and requires little additional cost. PhysFormer and PhysNet are both end-to-ent models. However, the number of parameters and computation of PhysFormer is much larger than the latter, which is not enough to reflect the advan- tages of lightweight and salltime. So EfficientPhys, MTTS-CAM and PhysNet are representative end-to-end models, and will compare the proposed method with them.
Many studies focus simplifying the pre- rrepprocessing process
2

---

Under review as a conference paper at ICLR 2023
2.2
FAST FOURIER CONVOLUTION
Chi et al. 2020)) proposed Fast Fourier Convolution (FFC), which can obtain an extensive range of receptive fields a shallow layer of the network. It improves accuracy on multiple tasks and datasets by replacing the convolutional layers in general convolutional networks with FFC layers. [Suvoroy al. (2022) applies FFC to wide range of image restoration, showing superior performance, especially for periodic images. Shchekotov et al. (2022) applied FFC to speech and audio signal denoising, significantly improving the baseline method and reducing the number of parameters. FFC is particularly sensitive to periodic signals and will significantly affect work that deals with periodic signals, such as audio noise reduction and rPPG. The difference between the above work and ours is that we do not model spatially and do not divide the time and frequency domain channels. We use simple time domain channels and skip them by residual connection. In this view, SEQ-rPPG does not model the video. We filter the noise from the ID sequence and extract the BVP signal.
METHOD
Handcrafted algorithms based on separate reflection components (Shafer 1985) assume that the BVP signal in the RGB image comes through a linear combination of different frequency rays, while the skin-reflected light contains specular reflection component and diffuse reflection component. In post-processing, stationary signals and noise are filtered, while periodic signals generated by fluctuations in hemoglobin concentration passed. Color pixel's dichromatic reflection model is shown below:
Cax+oY aCx + bCr,
where X and Y are power spectral density (PSD), PSD is the most commonly used method for heart rate extraction, which means that the rPPG signal can always be obtained by linear mapping of color space. In the early rPPG algorithm(Poh et al. 2010a b; de Haan & Jeanne 2013 Wang et al. 2017 Tulyakov et al., 2016), there are two steps from the RGB signal to the BVP signal. Firstly, based on linear dimensionality reduction method to obtain a 1-dimensional signal, and secondly, apply filtering post-processing in the time domain or frequency domain to filter out noise. In this paper, main steps (see Fig. D first, it linearly maps the RGB frames to the original signal sequence. Then uses 1D convolution to filter the original signal sequence and outputs the BVP waveform. The linear mapping is done inside the model, so this method does not require pre-processing. The parameters of the linear mapping module are obtained by learning, and after training, will have the appropriate weights to combine the RGB signals.
SEQ-rPPG also performs
8x8 facial
3.1 LINEAR MAPPING MODULE
Given RGB facial video input X E XXXXXXC first transpose it to X Nx ensure the channel is in the second dimension, then reshape it to E NCNWW X consists of N sets of RGB signals, the original signal is converted from RGB
WH Σ x (Ri, Gi, B,) +b;,
Y,
where m is a row vector, which maps RGB colors to real numbers, is the bias. The transformation of colors can be achieved by convolution without activation layers, it uses linear convolution layer with kernel size = 3 and stride = 3, linear mapping outputs multi-channel sequences, the number of channels depends on the number of convolution kernels.
3.2 SIGNAL PROCESSING MODULE
For the given original signal, it is usually filtered using several convolutional layers. This filtering is performed in the time domain, which means that adjacent signal points will be strongly correlated while distant signal points will be weakly correlated. For periodic signals, considering the correla- tion between similar periods helps filter noise from a wider range. From the signal processing point of view, classical BVP signal extraction usually requires filters to deal with the noise. Time-domain filters include Gaussian filters, moving average filters, and detrending, etc. Frequency domain fil- ters are commonly bandpass filters to extract signals specific frequencies. In summary, classical
3

---

Under review as a conference paper at ICLR 2023
C=64
N=1350
N-450
Reshape
Linear map
Signal process
H=8
K-a50
W-8
RGB sequence
Signal
B VP waveform
Raw facial frames
Stride=3
Spectral transform
Convolluionall filter
Spectral transform
wmm.m.
CCnnvolutional filter
kernel
Linear mapping module
Signal processing module
Figure 1: Overview of our framework.
methods contain alternating use of frequency domain filters and time domain filters. Inspired by the classical methods, we use the spectral transform layer to complete the processing in the fre- quency domain, while the traditional ID convolution is used for the time domain processing (see Fig. 2 For signal Y RV the spectral filtering layer first performs Real Fast Fourier Transform (RFFT) on each channel, obtain frequency domain signal Yrreg [N xC then is decomposed into a real part YReal and an imaginary part Yimg. they are combined on the channel |M x2C Then a convolution layer is applied to and re-decomposes the output into real and imaginary parts. It converted complex numbers and recovered the time domain signal by Inverse Real Fast Fourier Transform (IRFFT). The output signal is mixed with the original signal through the residual connection, and the number of channels remains constant throughout the process. The final signal is obtained by alternating spectral transform and ID convolution operation.
as YComb R
Conv ID L-BN-ReLU
C=128, K=8, S=1
Conv ID- D--BN-ReLU
C=64, K=15, S=1
Conv ID-IDD-BNN D-BBNReLUU
C=128, K=3, S=1
Conv ID-BN-ReLU
C=32, K=5, S=1
C=1, C=1,K=1, K=1, S=1
Flatten
RealFFTID
InvRcalFFTID
RealFFTID
InvRealFFTII
Conv
WW
$
C=64
Figure 2: Signal processing module.
DATASET
To collect the dataset, developed data capture software to obtain raw data from Logitech C930c webcam and Contech CMS50E Oximeter. The webcam has two capture modes, 1920* 1080 reso- lution, MJPG encoding, and 6400480 resolution, YUY2 encoding. Each mode has frame rate of
4

---

Under review as a conference paper at ICLR 2023
30fps. MJPG is common fast intra- frame compression format, and YUY2 is format for YUV images that uses bytes to store Y-components, U-component, and V-component. There are three video encoding formats, RAW RGB, MJPG, and H264. RAW RGB is the original lossless compressed RGB format, and H264 the common inter-frame compression format on the Internet. Multiple formats recorded simultaneously. Ground truth BVP waveform data is read from oximeter, the sampling rate is 20 per second. During data recording, subjects were required to com- plete series of tasks or watch video, and subjects were required keep their heads stable, so that they may have experienced larger head movements. After completing the assigned task, the subject will take short break, and the experimenter will assign him/her the next task. All 58 sub- jects (16 male and 42 female) are Chinese students, mostly master's students, and some girls may wear makeup. Excluding unavailable data due to formatting errors, incomplete tasks, or equipment failure, the video was available for over 30 hours (3.24M frames) of the video. Fig. 3 shows that the proposed dataset is well synchronized, where GT is the ground truth signal, and Video represents the BVP signal extracted from the video. More details of the self-build dataset can be found in Table and Fig. As shown in Fig. this dataset contains head movement, illumination variation, and expression change.
Video GT
Video
GT
BWV
BW
4
6
Time(S)
Time(S)
(a) UBFC-+PPG dataset
(b) Collected dataset
Figure 3: BVP signal extracted from UBFC-rPPG video and our collected video, where the collected dataset is highly synchronized.
Table I: Data collection workflows
Sub-dataset
Task induction video
Duration(S)
Camera codec
Video codec
Resolution
Relaxed
120
Relaxed Dark
120
rPPG
YUY2
RGB,MJPG,H26A
640x480
Play game
120
Read an article
120
Natural scenery
120
Puzzle game Comedy
180
120
Emotion
MJPG
MJPG,H264
1920x1080
Illusion picture
Academic paper
Video about yawn
60
Rideo-based learning
240
Engagement
satbbokk-bssdd learning
480
MJPG
MJPG.H264
1920x1080
Watch public class
420
5 EXPERIMENTS
5.1 TRAINING DATA & TESTING DATA
Since our model cannot be trained effectively on non-synchronized datasets, we only trained it on the collected dataset and PURE dttaset(Stricker et al. 2014), excluding small amount of data with
5

---

Under review as a conference paper at ICLR 2023
Figure 4: Overview of our dataset, subjects' faces can move unconstrained and emotions are induced during tasks.
a low frame rate or wrong format due to insufficient performance during recording. Although we acquired the data in multiple video encoding formats, only the MJPG format was used for the entire training and testing. For the self-build dataset, we randomly divided the 58 subjects into five folds, the first four folds were used as the training set, and the last fold was used as the testing set, and all videos of the same subject would only appear in the same fold. The test is divided into two parts, first on the whole testing set and then the rPPG subset, which contains more complex scenarios with motion. Details about the dataset partitioning are saved in a CSV file and made public along with the dataset. Models were also tested the UBFC-IPGG(Bobir UBFC-rPPG includes 42 videas from 42 subjects of approximately I minute each, all recorded at resolution of 640x480, frame rate of 30 fps, RAW RGB format, and all faces were kept stable. PURE includes 59 one-minute videos from 10 subjects, including 6 types of head movements, all recorded at resolution of 640x480, frame rate of 30 fps, in RAW RGB format.
al. 2019) and PURE public dataset.
5.2 IMPLEMENTATION & EXPERIMENT DETAILS.
To compare with Liu et al. 2020) and Yu al. (2019) on the collected data set, we reproduced the TS-CAN directly using the source code provided by Liu et al. (2020). Since Tensorflow per- formance is usually higher than Pytorch, to be fair in performance tests, we implemented PhysNet
and EfficientPhys using Tensorflow according to the details provided in Yu et al. (2019): Liu et al. 2021
We have three models: SEQ-FT with alternating con- volution in the frequency and time domains, SEQ-T with convolution in the time domain only, and the
RGB sequence
RGB sequence
streamlined SEQ-tiny (see Fig (5) All models are trained and tested in the same Tensorflow environ- ment, the version is 2.6.0, compiled with cuda 11.6, cudnn 8.3, AVX2. Hardware platforms include Nvidia Geforce RTX 3080 10G (GPU), AMD Ryzen9 5950x 16-core (CPU), ARM Cortex A72 4-core (CPU), when tested ARM platform, also used Tensorflow 2.6.0, but does not include any acceleration support for GPUs or x86 CPUs. To train efficiently, we cut the video us- ing a moving window with a step size of 3 seconds and used mediapipe to perform face detection in the mid- dle of the frame (sometimes other people appeared in the background but were already processed). Finally, only the facial images were fed to the model.
Conv ID C=64, K=3, S=3
SeparableConv ID C=16, K=3, S=3
Conv ConvID-BN-ReLJ. C=64, K=15, S=1
ConvID-BN-Rad C=16, K=5, S=5
ConvID-BN-ReLL.
C=64, K= S=1
ConvID-BN ReLU C=8, K=5, S=1
ConvIID-BN-Rel.. C=32, K=5, S=1
ConvIDTranspose BN-ReL. C=4, K=5, S=5
ConvID-BN-ReLJU C=32, K=35,K=5, S=1
ConvlD C=1, S=1
ConvlD K=l, S=1
Flatten to BVP
Flatten to BVP
(a) SEQ-T
(b) (b)SEQiy
Although Liu et al. (0220); Yu et al. (201)) Liu et al. (202T) claims to be rre-processing-tres model, face
Figure 5: Overview of SEQ-T & SEQ-tiny.
detection had to be performed because of the large movements in the collected dataset. All models used the same data during training and testing, differing only in resolution. collected dataset, we randomly selected a part of the segments with interval skipping half of the frames and did the same for the BVP waveform, which finally made the heart rate twice as high. The additional data accounted for about 25% and were evenly distributed in the training set. TS- CAN uses the Nadam optimizer with default parameters, MAE loss, batch size=128. PhysNet uses the SGD optimizer with learning rate=0.005, NP loss, batch size=32. our model uses the adam op- timizer with default parameters, MSE loss, batch sizc=32. Classical methods and TS-CAN used a 0.75 to 2.5 Hz bandpass filters in the test. Neither our models nor PhysNet applied filters. CHROM,
addition, due the lack of enough high heart rate samples in the

---

Under review as a conference paper at ICLR 2023
POS and ICA(de Haan & Jeanne 2013; Wang et al. 2017 Poh et al. 2010b) the code provided
in Boocignone et al. (2022) and Boccignone et al. (2020). The final heart rate is extracted from the peak of the signal power spectrum, ranging from 0.5 to 4 Hz. In calculating the error and Pearson correlation coefficients, we used a 30-second moving average window with 1-second step.
We evaluated the accuracy and speed of the model separately, where the accuracy metrics include Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Pearson correlation coefficient (p). For the speed test, we calculated the number of parameters, FLOPs (see Table 2), and memory usage of the model and tested the inference speed (ms) on desktop GPU, desktop CPU, and mobile CPU. We recorded the hardware load to indicate the efficiency of the model when the model was running at the highest speed.
Table Theoretical parameters and FLOPs
Model
Parameters
FLOPs
SEQ-tiny
3,46K
1.49M
SEQ-T
152K
136M
SEQ-FT
266K
157M
TS-CAN
533K
6.68G
PhysNet
770K
6,95G
EfficientPhys-C
2.16M
29.5G
5.3 EXPERIMENTAL RESULTS
The extensive experimental results are presented in Tables 3 and 5 The rPPG subset is more difficult because has more light changes and facial activity. SEQ-FT achieved the best accuracy on the collected dataset, with second place going to EfficientPhys C-C. The results show that SEQ-FT more accurate than SEQ-T, and the spectral transformation is effective.
We compared the results of different models tested on UBFC-rPPG, which were trained on different training sets. Our models perform similarly, and all have high accuracy. Although the difference in the number of SEQ-FT and SEQ-tiny parameters is very large, their accuracy is similar, which may be caused by the stable head and stable lighting environment of UBFC-IPPG. TS-CAN and PhysNet are trained better the collected dataset than other datasets. In particular, the results the collected dataset outperform those trained on UBFC-rPPG, indicating that a high-quality training set helps improve the model accuracy. Among all the cross-dataset results on UBFC-rPPG, the best is TS-CAN trained on the collected dataset, and the second best PhysNet trained on the collected dataset. On the PURE dataset(Table 5). EfficientPhys-CC achieves the best, followed by TS-CAN. Notably, compared to training on UBFC-rPPG, the improvement of EfficientPhys-C by training on the collected dataset significant, with MAE dropping from 5.90 to 1.82.
As observed from the results, our model has very high accuracy on the collected dataset, so it is more suitable for educational scenarios. Our model competitive with small models like TS-CAN on two public datasets, UBFC-IPPG and PURE, and our model parameters and FLOPs are much smaller. There still gap compared larger models like EfficientPhys. The results of swapping the training set and the test set are interesting. Cross-dataset evaluations are usually trained on PURE and tested on UBFC-rPPG, but few papers have reversed them. The results suggest that the models trained on UBFC-IPPG may lack generalizability and perform poorly on PURE. The good news is that our collected dataset can solve this problem. The models trained on the collected dataset can be applied to different test sets.
Table 3: Results on collected dataset
Entire dataset
rPPG subset
Method
MAE]
RMSE
pf
MAEL
RMSEL
pT
SEQ-tiny
3.57
6,98
0.72
5.70
9.82
0.60
SEQ-T
1.77
4.69
0.85
3.16
6.69
0.72
SEQ-FT
0.95
2.12
0.96
1.55
3.29
0.89
TS-CAN
1.39
2.47
0.95
2.44
3.60
0.86
PhysNet
3.13
6,61
0.78
4.94
9.80
0.56
EtficiemPhys--C
1.26
2.17
0.96
1.91
2.70
0.94
CHROM
7.73
10.8
0.43
8.50
10.0
0.12
POS
3.91
6.03
0.70
5.10
7.10
0.43
ICA
11.4
13.5
0.29
12.2
14.7
0.17
7

---

Under review as a conference paper at ICLR 2023
Table Cross-dataset evaluation UBFC-IPPG
Method
Training set
MAEt
RMSEL
PT
SEQ-tiny
Collected
0.82
1.21
0.99
SEQ-tiny
PURE
6.66
9.78
0.68
SEQ-T
Collected
0.93
1.47
0.99
SEQ-T
PURE
4.36
7.84
0.77
SEQ-FT
Collected
0.81
1.21
0.99
SEQ-FT
PURE
2.45
4.07
0.95
PhysNet
Collected
0.72
1.16
0.99
PhysNet
PURE
1.99
4.49
0.97
TS-CAN
Collected
0.69
0.95
0.99
TS-CAN
PURE
1.47
2.31
0.99
Efficienthyyse Phys-C
Collected
0.87
1.48
0.99
EfficientPhys-C
PURE
2.04
3.06
0.99
Table Cross-dataset evaluation on PURE
Method
Training
MAEJ
RMSEL
PT
SEQ-tiny
Collected
5.80
12.0
0.86
SEQ-T
Collected
4.34
9.40
0.91
SEQ-FT
Collected
3.55
8.86
0.92
PhysNet
Collected
7.83
12.1
0.89
PhysNet
UBFC
8.39
19.2
0.71
TS-CAN
Collected
3.06
6.27
0.95
TS-CAN
UBFC
5.75
16.3
0.74
Efficient Phys-C
Collected
1.82
3.91
0.99
EfficientPhyss Phys-C
UBFC
5.90
9.75
0.92
5.4
COMPUTATION COST
Table 6: Computation cost on different platforms
Memory usage
Desktop GPU
Desktop CPU
Mobile CPU
Method
Batch
Model (KB)*
Input (KB)
Time (ms)*
Time (ms)
Time, (ms)
Loadt
Load
Load
SEQ-tiny
450
13.8
346
1.1
97%
0.7
12%
6.3
40%
SEQ-T
450
608
346
1.7
99%
2.4
50%
34
72%
SEQ-FT
450
1064
346
2.2
99%
12
17%
146
61%
TS-CAN
128
2132
1991
9.1
99%
79
77%
1260
91%
PhysNet
128
3080
1573
9.7
99%
48
79%
1290
89%
EfficientPhys-C
128
8656
7963
11.0
99%
271
68%
8530
93%
We also tested the computation cost of the model on different platforms (see Table 6). Unlike the time spent per frame in Liu et al.] (2020). we caleulated the time spent to process whole batch. In past work, little attention was paid to the input size. Usually, the input is resized, and float32 numerical matrix obtained. If the input size is large, then the memory usage of the input data may be higher than the model itself. The memory usage of the model is a theoretical minimum based on the number of parameters, which will be smaller than actual value. We tested the average inference time of the model on large data as the time-use result, device load is also concerned. The slow inference speed of the model on some devices may be due to the inability to use computational resources efficiently, resulting in low load state the device. Although both low load and high computation cost result slower speeds, the energy consumed and heat generated are not the same. Smaller computation cost means lower heat generation and more energy efficiency, which is espe-
8

---

Under review as a conference paper at ICLR 2023
cially important cattery-powered devices. Our model has significant advantage in speed, being much faster and consuming much less power than its predecessor. These models can be divided into three classes depending ARM mobile CPUs' performance. (: Can run in real-time platforms but not on mobile devices, such as EfficientPhys. 2: Can run in real-time on mobile de- vices, but the computation cost is significant and increases heat dissipation and battery loads, such as PhysNet and TS-CAN. 3 Can run smoothly on mobile devices, and the computation cost is tiny, such the proposed method.
desktop
5.5 VISUALIZATION & DISCUSSION
To visually verify the effectiveness of the proposed method, SEQ-FT chosen for visualization, and we randomly selected a video to feed into the model, and the intermediate feature maps were captured and used for visualization whenever possible. Each feature map can be viewed as a multi- channel signal sequence, and its Power Spectral Density (PSD) is calculated for each channel. The 0.5-2.5 Hz portion intercepted for visualization. significant heart rate peak observed in the output from the linear mapping. However, a considerable amount of noise power is also present in PSD. As the features go deeper, the noise contained, and some details start to appear, such as subpeaks in the high-frequency region. As shown in Fig 6 the final output PSD is very close to the ground truth. Through feature visualization, our model is well interpreted, and our visualization of temporal modeling demonstrates the role of convolution networks in another dimension than Liu ctt al. (2020) and Niu et al. (2020a)'s 's focus on spatial modeling.
P K.
PSDIWIHD
25
Frequency
Froquency(Hzz)
Frequency[Iz))
(a) Linear mapping outputs
(b) First spectral transform.
(c) domain conv.
(SD(WHH))
PSD(WHH)))
PSD(W/H2)
Froqueucy(H2)
Frequency(HL))
Frequency(H2)
(d) Second spectral transform.
(c) Second domain Conv.
(f) Final output and ground truth
Figure 6: Power Spectral Density(PSD) of the feature maps.
CONCLUSIONS
In terms of accuracy, our model performs similarly to or better than existing models; in terms of speed, our model reaches several times more than current models and can run effectively on mobile CPUs. On Raspberry Pi 4B, our highest accuracy model inference takes 146ms, and the fastest model takes 6.3ms, compared with TS-CAN's 1260ms and EfficientPhys' 8530ms. However, SEQ- FT does not perform as expected on CPUs, and it is worth investigating how to make the spectral transform run more efficiently on CPUs. The quality of our dataset is better than UBFC-IPPG and PURE. The accuracy of existing models can be improved significantly by training the collected dataset, e.g., the MAE of TS-CAN decreases from 1.47 (trained on PURE) to 0.69; the MAE of PhysNet decreases from 1.99 (trained on PURE) to 0.72, and expect the collected dataset to be baseline for training sets. However, the proposed method is sensitive to the random shift present in the dataset, and we will try to deal with this issue in the future.

---

Under review as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
Although the steps required to reproduce the results have been described in this paper, factors that significantly affect the results need be highlighted in this section. This model has not been finely tuned to the parameters, and its accuracy is not very sensitive to the parameters but to the training set. When working with datasets, there were some key operations: First, we implemented face detection and tracking. However, face detection frames are usually difficult to keep stable. We added a displacement threshold of size 25 pixels when implementing face tracking. Any movement at distance less than the threshold will be filtered, and greater than the threshold will move the detection frame suddenly, thus avoiding introducing periodic noises. Secondly, when making the training set, we made additional high heart rate samples with heart rates in the 120-180, representing about 25% of the total number of samples. The samples were obtained by skipping half of the frames at intervals, so the heart rate would be twice high, which is exceptionally important for training the model to correctly identify the high heart rate samples; if enough high heart rate samples are not correctly generated in the training set, then great errors may occur by chance when estimating the high heart rate test set samples. Finally, we ensured that the BVP signal in the training set was strictly aligned with the frames by the recorded UNIX timestamps. Our model does not require facial segmentation, but tracking using facial detection is necessary due to the uncertainty of the head position in the collected dataset. The above steps were applied to all models we implemented.
REFERENCES
Serge Bobbia, Richard Macwan, Yannick Benezeth, Alamin Mansouri, and Julien Dubois. Unsu- pervised skin tissue segmentation for remote photoplethysmographyy Pattern Recognition Letters, 124:82-90, 2019.
Giuseppe Boccignone, Donatello Conte, Vittorio Cuculo, Alessandro D' A'melio Giuliano Grossi, and Raffaella Lanzarotti. An open framework for remote-PPG methods and their assessment. IEEE Access, pp. 1-1, 2020. doi: 10. 1000///cess URL httpt::tp////// /rrg 10.1109/access 2020 3040936
Giuseppe Boccignone, Donatello Conte, Vittorio Cuculo, Alessandro D' D'Amelio, Giuliano Grossi, Raffaella Lanzarotti, and Edoardo Mortara. pyvhr: python framework for remote photoplethys- mography. PeerJ Computer Science, 8:e929, 2022.
Deivid Monsaa-M Yannick Benezeth, and Johel Miteran. Rtrppg: An ultra light 3denn for real-time remote photoplethysmography In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition (CVPR) Workshops, pp. 2146-2154, June 2022.
Weixuan Chen and Daniel McDuff. Deepphys: Video-based physiological measurement using con- volutional attention networks. In Proceedings of the European Conference on Computer Vision
(ECCV), September 2018
Lu Chi, Borui Jiang, and Yadong Mu. Fast fourier convolution. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neural In- Curran Associates, Inc.,
formation Processing Systems, volume 33, pp. newrips....... 2020.
URL https://prroeediggs.r hewpeeeeeeddnnn.n.eiiiii.s..ppss
Paper pdf
J. Comas, A. Ruiz, and F. Sukno. Efficiint remote photoplethysmography with temporal deriva- tive modules and time-shift invariant loss. In 2022 JEFF/CVF Conference on Computer Vi- sion and Pattern Recognition Workshops (CVPRW), pp. 2181-2190, Los Alamitos, CA, USA, jun 2022, IEEE Computer Society. doi: 10. LLLU00//VVPRW3333222222.000
URL https:
7/doi
2022. 00237
Gerard de Haan and Vincent Jeanne. Robust pulse rate from chrominance-based rppg. IEEE Transactions on Biomedical Engineering, 60(10))2878-288 2013. doi: 10.1109/TBMF.20133 2266196.
Xin Liu, Josh Fromm, Shwetak Patel, and Daniel McDuff. Multi-task temporal shift attention net- works for on-device contactless vitals measurement, In H. Larochelle, M. Ranzato, R. Hadsell,
10

---

Under review as a conference paper at ICLR 2023
M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 19400-19411. Curran Associates, Inc., 2020. URUAttpss//Drooeddngg... nww..iip...ngnnnn...............
Xin Liu, Brian L. Hill, Ziheng Jiang, Shwetak Patel, and Daniel McDuff. Efficientphys: Enabling simple, fast and accurate camera-based vitals measurement, 2021. URL https://arxiv.
erq/abs/De 22100.210.. 04447
Hao Lu, Hu Han, and S. Kevin Zhou. Dual-gan: Joint bvp and noise modeling for remote physiolog- ical measurement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12404-12413, June 2021.
Daniel McDuff, Miah Wander, Xin Liu, Brian I. Hill, Javier Hernandez, Jonathan Lester, and Tadas Baltrusaitis. Scamps: Synthetics for camera measurement physiological signals. arXiv preprint
07Xi::2206.04197, 2022.
Xuesong Niu, Shiguang Shan, Hu Han, and Xilin Chen. Rhythmnet: End-to-end tnd heart rate estima- tion from face via spatial-temporal representation. IEEE Transactions on Image Processing, 29: 2409-2423, 2020a. doi: 10.1 109/T1P.2019.294790
Xuesong Niu, Zitong Yu, Hu Han, Xiaobai Li, Shiguang Shan, and Guoying Zhao. Video-based remote physiological measurement via cross-verified feature disentangling. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.). Computer Vision - ECCV 2020, pp- 295-310, Cham, 2020b. Springer International Publishing. ISBN 978-3-030-58536-5
Ming-Zher Poh, Daniel McDuff, and Rosalind W Picard. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Optics express, 18710):00622 10774, 2010a.
Ming- Zher Poh, Daniel J McDuff, and Rosalind W Picard. Advancements in noncontact, multi- parameter physiological measurements using webcam. IEEE transactions on biomedical engi- neering, 58(1):7-11, 2010b.
Steven A. Shafer. Using color separate reflection components. Color Research and Applica- fion, 10:210-218, 1985. doi: 10. 10.1002/col.500010000 URL https:://doi http///ooorrg/....o/o 1002/ col. 5080100494
Ivan Shchekotov, Pavel Andreev, Oleg Ivanov, Aibek Alanov, and Dmitry Vetrov. Ffe-se: Fast fourier convolution for speech enhancement, 2022.
03042
Rencheng Song, Huan Chen, Juan Cheng, Chang Li, Yu Liu, and Xun Chen. Pulsegan: Learning to generate realistic pulse waveforms in remote photoplethysmography. IEEE Journal of Biomedical and Health Informatics, 25(5):1373-1384, 2021. doi: 10. 2109///B...2...15111111
Ronny Stricker, Steffen Müller, and Mirst-Michael Gross. Non-contact video-based pulse rate mea- surement on mobile service robot. In The 23rd IEFE International Symposium on Robot and Human Interactive Communication, pp. 1056-1062, 2014. doi: 10.1 1..11090OO.A.. 2.66699...
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong. Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. In Proceedings of the IEEE/VVF Winter Conference on Applications of Computer Vision (WACV), pp. 2149-2159, Jan- uary 2022.
Chihiro Takano and Yuji Ohta. Heart rate measurement based time-lapse image. Medi- cal Engineering & Physics, 29(8):853-885 857, 2007. ISSN 1350-4533. doi: https://doi.rr//I 200.: 20..00000..000 000066...00...... URL httpt::////// //w:://wwwww.. sci encedirects wtw.eeettcccceeeem ccom/cc ccom/ccience. ence/ 901
10166j.medengnhy
articlle/p11/11104553300
Sergey Tulyakov, Xavier Alameda-Pinedae Elisa Ricci, Lijun Yin, Jeffrey F. Cohn, and Nicu Sebe. Self-adaptive matrix completion for heart rate estimation from face videos under realistic con- ditions. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 2396-2404, 2016. doi: 10.1 10.1109/CVP.2201..665

---

Under review as a conference paper at ICLR 2023
Wim Verkruysse, Lars O Svaasand, and Stuart Nelson. Remote plethysmographic imaging using ambient light. Opt. Express, 16(26):21434 21445, Dee 2008. doi: 10. 1364/164.EE... 16.021434. URL owt.opiii...... wttp://gg////.taatteeeeba abstract ww 1---1166-226-6-6
ce
Wenjin Wang, Albertus C. den Brinker, Sander Stuijk, and Gerard de Haan. Algorithmic principles of remote ppg. IEEE Transactions on Biomedical Engineering, 64(7):147--1491, 2017. doi: N0.1109/TBME222166260222
Zitong Yu, Xiaobai Li, and Guoying Zhao. Remote photoplethysmographh signal measurement from facial videos using spatio-temporal networks. arXiv preprint SEBi::9905.02411 2019.
Zitong Yu, Yuming Shen, Jingang Shi, Hengshuang Zhao, Philip Torr, and Guoying Zhao. Phys- former: Facial video-based physiological measurement with temporal difference transformer. In CVPR, 2022.
A APPENDIX
A.I PARAMETER DISCUSSION AND ABLATION EXPERIMENTS
Our approach is sensitive to the length of the input frame sequence due to its focus modeling in the time dimension. We trained SEQ-FT on the collected dataset and tested on the collected dataset and UBFC-IPPG(Table 77. From the results, longer sequence improve the accuracy. However, we do not want too long input sequences to cause too much delay, so 15 seconds(450 frames) is a suitable value.
Table 7: Effect different input sequence lengths on accuracy
Mean Absolute Error
Length
Collected
UBFC-rPPG
128
3.22
12.6
256
2.21
3.24
384
1.09
1.57
450
0.95
0.81
512
0.81
0.77
Our model ignores spatial information, higher resolution does not help to improve the accuracy, but greatly increases the parameters. We trained SEQ-FT on the collected dataset, and tested on the collected dataset and UBFCxPPG(Table 8). The results show that using resolution of 8x8 is sufficient.
Table 8: The effect of different resolutions on accuracy
Mean Absolute Error.
Resolution
Collected
UBFC-rPPG
4x4
3.34
7.49
8x8
0.95
0.81
16x16
0.96
0.98
32x32
1.05
0.92
Our model uses time-domain convolution and spectral transformation. and ablation experiments show that the combination these two is optimal. All models were trained on the collected dataset, and tested on the collected dataset and (BFC-IPPG((Tblee)
A.2 PRIVACY
If the rPPG method is applied to a commercial application, the server expects to capture the user's facial image optimize the user experience and improve the algorithm. However, the user's facial
12

---

Under review as a conference paper at ICLR 2023
Table 9: Effect of different model structure and number of layers on accuracy
MAE collected dataset
MAE UBFC-rPPGI
Layers
F
T
F&T
F
T
F&T2
3.73
3.85
2.61
7.81
2
2.84
1.86
1.60
0.97
3.74
2.14
3
2.18
1.80
1.08
2.31
1.77
1.44
0.95
0.93
1.90
0.81
5
1,40
1.43
15.3
1.45
1.65
1.21
1.08
14.8
1.32
1.49
Spectral transformation in the frequency domain 1D convolution in the time domain
image involves privacy issues, and the higher resolution, the more private information it contains (see 7). so using low-uusolution images helps to protect privacy. Besides, low-rcsolution images reduce network usage during uploading and improve the experience. Our model uses 8x8 inputs, which implies strong privacy.
128x128
(b) 64x64
(c) 32x32
8x8
Figure 7: Facial images in different resolutions.
A.3 Low LATENCY
Since our model uses a long sequence of 450 frames of input, it usually requires 15 seconds to obtain the BVP signal. Compared with models using 128 frames of input, model seems to have to wait longer. However, in engineering, this can be optimized. For example, for a 60 frames input, we simply repeat it eight times get 480 frames of input and intercept 450 of them. In the output signal, the BVP waveform can be obtained by intercepting 60 frames from the middle at the corresponding position, and this operation greatly reduces the delay (see Fig 8
A.4 SIGNAL QUALITY
TS-CAN must use bandpass filter to improve signal quality. For PhysNet, additional filters not required. the use of additional filters has almost no effect on SEQ-FT (see Fig 9).
A.5 MORE EFFICIENT PRE-PROCESSING
Simple pre-processing can be eliminated by reasonable engincering optimization. In the original article, the pre-processing of TS-CAN was done by NumPy, and we tried to do the same using Tensorflow and move this operation inside the model. This means that we only need input the original video after deploying the model and do not need to pay attention to the specific implemen- tation of pre-processing, which helps be implemented in non-python environments (e.g. mobile web), this operation does not affect the accuracy and makes full use of Tensorflow's powerful matrix operations.
Approach of the original literature motion, appearance pre_process (vid) Pre-processing ng with numpy
13

---

Under review as a conference paper at ICLR 2023
15
15s video
2s video
10
0.5
BVP
-0.5
-1.0
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
200
Time(S)
Figure 8: Extract the BVP signal from seconds 15 seconds of video in the duration
Ground truth
15 CAN RAW
-1
15, CAN filtered
PhysNet
SEC-FT
Time[5]
Figure 9: BVP Signal from different model
BVP TS CAN (Imotion, appearance
def call (self, x)
x_ (x[1:] x(:=1))/(((1:)>xx(::-
14

---

Under review as a conference paper at ICLR 2023
x_ tf. reduce mean (x_, (xis= ))) /t)/ttfmath...tttmmmmmm www.cattth..ueeeeeeeeeeeee (x_,
axis= })
moticn, appearance concat ( [xx,, tf. zeros ([1, 4size, 3])],
axis=0), tf. expeepaad__ia dims (tf. www.ceddddeeeeee mocoe__oaannii (x, axis= )), axis=0)
d1, rl = dalf.difffinee input (meticn), self. www.:raafffnu (appearance)
better approach, closer to end-to-end
BVP = IS_CAN (vid)
15


Reviews:
[{"review_id": "zJy6RQgOIj_", "reviewer": "Reviewer_j11H", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a simple method, followed by a dataset for remote PPG/BVP monitoring. The proposed method is a simple neural model with reasonable results, and the dataset has 58 participants.", "summary_of_the_review": "Please see my review above.", "strength_and_weaknesses": "The main strength of the paper is the dataset component of the work. However, the paper, unfortunately, has a number of weaknesses that needs to be addressed. These weaknesses are as follows:\n\n1- The writing, presentation, and editing are not very polished. \n\n2- The paper is very hard to follow. I find it hard to put my finger on what exactly the contributions of the paper are.\n\n3- The proposed method is very simple and trivial, and does not provide any new \"representation learning\" insights.\n\n4- The paper cites several well-known datasets in the area, including MAHNOB-HCI, VIPL-HR, and others, some of which have more participants and are more diverse. It is therefore not clear why a new dataset was needed.\n\n5- The dataset is not very diverse in terms of participants, and many details are missing, e.g. M/F ratio, age range/SD, etc\n\n6- There is no confirmation that ethics approval has been secure, which is really important.\n\n7- The dataset is not made public (and doesn't seem to be planned to go public). So I am not sure if it can be considered a contribution to the area. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "wvIOZS-BHHz", "reviewer": "Reviewer_DUL3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an approach for extracting the blood volume pulse signal via photoplethysmography from a video.  This use case of computer vision has several positive applications, that could help make scalable physiological sensing possible.  The authors do a thorough job summarizing the existing literature, including many of the most recent papers.  While the summary could be made a little more readable, I think it is a well synthesized introduction to the paper on whole.  The proposed method uses existing but computationally efficient neural layers to extract a PPG signal from the video.  The results overall appear reasonable and the method performs well compared to the baselines in the comparisons that are given.\n", "summary_of_the_review": "Overall, I believe this application is very exciting.  I think that the proposed method has some novelty and focusing on fast/computationally light algorithms is good.  The paper has the makings of a good contribution that I’d like to support.  However, I feel like the paper as it is currently falls short in empirical support for their method and theoretical support. If the authors can address all these points above - I would be willing to increase my rating.", "strength_and_weaknesses": "The authors correctly highlight deficiencies with the existing public datasets which are invariably small and limited in diversity. They present a new dataset that they collected which is arguably larger than the existing data. However, their dataset is also limited in diversity with regard to skin types and age, based on the description. I think their claims that their dataset is particularly large in comparison to existing datasets is not really true - it is comparable in order of magnitude. I think that those claims should be tempered a bit. It was not very clear to me whether the dataset will be made available as a benchmark set?  Please clarify - making the contribution of the dataset a central claim in the paper without releasing it is frustrating to readers as it doesn’t help to advance the communities’ efforts, beyond the insights gained from a single paper.\n\nThe authors state that for some existing methods the \"input batch size is 128 frames, which means it has a latency of over 1s. To get a real-time BVP signal, the latency should be as small as possible.\" However, it is not that clear to me in what application a delay of 1 or even a couple of seconds would be a problem.  In heart rate estimation a period of at least 5 seconds would be need at least any way.  Can the authors clarify why a latency of 1 second is significant here?  I can see the memory or compute constraints with a longer temporal window being more significant which may better support the authors arguments here. \n\nThe approach to solving the problem does appear to be novel.  It is interesting that the authors collapse the input to RGB vectors thereby losing spatial information, which is often useful for segmentation.  I would question how sensitive their method is the any face detection or segmentation step they perform. I would appreciate a comment on this, again with little visibility on the dataset they collected it is hard for a reader to ascertain this.  To add to this point, the paper is short on experimental details and while the supplementary material contains model implementations, it does not actually include end-to-end training code which is disappointing as their are often details in steps prior to, or following, a model that impact performance (e.g., segmentation steps, filtering steps etc.). I would appreciate the authors comments on any additional details that could be provided to help ensure reproducibility.\n\nSection 5:  It is still a bit unclear to me after reviewing the paper why the proposed method cannot be trained on other datasets.  Yes, some datasets do not have very highly synchronized PPG and video, but some do.  In my experience, for example, training models with PURE or SCAMPS is possible.  Can the authors be more precise about why training with those existing datasets isn’t possible? I would appreciate a clearer indication or examples to illustrate this.  \nAnd in Table 4 - why wasn’t EfficientPhys, PulseGAN or DualGAN trained on the collected dataset?  Is that because those models were not implemented and the results taken from prior work?\n\nThe above point applied to training, for testing it would seem that the authors could use any dataset.  Why did the authors only choose to test on UBFC and not any of the other benchmark datasets?  This is a significant weakness of the work as almost all other papers manage to test on at least 2-3 public datasets.  Could the authors produce results on these other datasets? \n\nIn Section 5 the authors write: Unlike the time spent per frame in Liu et al. (2020), we calculated the time spent to process a whole batch, which is more reflective of the latency caused by the computation. However, while this is true about latency, it doesn’t reflect the computational cost which I think is more the point (see the comment above about the difference between latency and computational cost). I don’t think latency is really a bottleneck in this application, but computation is. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "j-DaD3Q0Dd", "reviewer": "Reviewer_V5oM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new dataset for blood volume pulse (BVP) estimation which addresses some shortcomings of existing datasets. A method to estimate BVP from RGB frames is also introduced. Results on the collected dataset and other publicly available datasets are presented and the proposed approach achieves competitive performance without requiring complex pre-processing.\n\n", "summary_of_the_review": "There are several weaknesses as explained above which need to be addressed before the paper can be considered for acceptance.", "strength_and_weaknesses": "Strengths\n\nA new dataset is presented which will be a nice contribution (if it becomes available).\n\nWeaknesses\n\nA major weakness in the paper is that it’s not easy to follow the discussion. There are no references to tables or figures, so the reader is left searching for the corresponding tables/figures.\n\nThe introduction reads more like a review of existing methodologies, it’s hard for the reader to understand what the main contributions are since they are described in different paragraphs. It would be good to have a paragraph which lists the main contributions. For example the end of the last paragraph could be extended to summarise all contributions.  \n\nAn ablation study is missing in the paper. This would reveal the impact of each component.\nFor example, the authors use an image of 8 by 8 in order to alleviate privacy concerns. However, this is a very low resolution and it would be good to test the performance of the proposed approach as a function of the input resolution. Another example is studying the impact of the number of convolutional and spectral layers and the impact of the input sequence length.\n\nOne of the main contributions of this paper is the introduction of a new dataset, however only a brief description of the dataset is given and it is not mentioned if the dataset will become publicly available. The latter Is quite important as if the dataset is not released it diminishes the paper’s contribution. It would also be useful to add more details in the dataset description, e.g., age/gender distribution, did all subjects do all tasks, total number of videos and duration of the entire dataset.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject