EfficientPhys: Enabling Simple, Fast, and Accurate Camera-Based Vitals Measurement
Authors: Xin Liu
Brian L. Hill
Ziheng Jiang
Shwetak Patel
Daniel McDuff
Venue: ICLR 2022 Submitted (2022)
Keywords: Computer Vision
Healthcare
Deep Learning

Abstract: Camera-based physiological measurement is a growing field with neural models providing state-the-art-performance. Prior research have explored various "end-to-end'' models; however these methods still require several preprocessing steps. These additional operations are often non-trivial to implement making replication and deployment difficult and can even have a higher computational budget than the "core'' network itself. In this paper, we propose two novel and efficient neural models for camera-based physiological measurement called EfficientPhys that remove the need for face detection, segmentation, normalization, color space transformation or any other preprocessing steps. Using an input of raw video frames, our models achieve state-of-the-art accuracy on three public datasets. We show that this is the case whether using a transformer or convolutional backbone. We further evaluate the latency of the proposed networks and show that our most light weight network also achieves a 33\% improvement in efficiency.


Under review as a conference paper at ICLR 2022
EFFICIENTPHYS: ENABLING SIMPLE, FAST AND AC- CURATE CAMERA-BASED CARDIAC MEASUREMENT
Anonymous authors
Paper under double-blind review
ABSTRACT
Camera-based physiological measurement is a growing field with neural models providing state-the-ar tthe-art--pr art-performance. Prior research have explored various "end- to-end" models; however these methods still require several preprocessing steps. These additional operations are often non-trivial to implement making replication and deployment difficult and can even have higher computational budget than the "core" network itself. In this paper, we propose two novel and efficient neural models for camera-based physiological measurement called EfficientPhys that remove the need for face detection, segmentation, normalization, color space transformation or any other preprocessing steps. Using an input of raw video frames, our models achieve state-of-the-ar accuracy on three public datasets. We show that this is the case whether using transformer convolutional backbone. We further evaluate the latency of the proposed networks and show that our most light weight network also achieves a 33% improvement in efficiency.
INTRODUCTION
Camera- basera physiological measurement is non-contact approach for capturing cardiac signals via light reflected from the body. The most common such signal is the blood volume pulse (BVP) measured via the photopltthysmogram (PPG). From this, heart rate (Takano & Ohta 2007 Verkruysse ct al. 2008), , respiration rate (Poh et al. 2010) and pulse transit times Shao et al. 20 can be derived. Furthermore, there is promising evidence that the PPG signals be be used to measure signs of arterial disease Takazawa et al. (1998). Neural models are the current state-of-the-art in this domain (Chen & McDuff 2018a| Liu et al. 2020a 2021d). These networks can learn strong feature representations and effectively disentangle the subtle changes in pixels due to underlying physiological processes from those due to body motions, lighting changes and other sources of "noise".
While prior research has framed architectures "end-to-end" methods, those that achieve state-of- the-art performance actually require several preprocessing steps before data is input into the network. For example, Chen & Mcfuff (2018a) and Liu al. (2020a) use hand-crafted normalized difference frames and normalized appearance frames as imput to their convolutional attention network. NNu et al.) (2020) and [u et al. ( use complex schema create feature maps called "MST "MSTmap".. This process includes facial landmark detection, extractions of several regions of interest (ROI) using these landmarks, and then averaging pixel values in both the RGB and YUV color spaces.
These preprocessing steps have several drawbacks: 1) They make assumptions about optimal nor- malization or representation without allowing the network learn these features in a data-driven manner. 2) They are computationally costly and in many cases add a significant number of operations the video processing pipeline. There are several reasons why we would prefer to run camera-base physiological sensing on-device: preserving privacy, analyzing raw video (i.c., not compressed) and saving data costs and bandwidth. Therefore, any additional computation needs to be justified by improving model accuracy, otherwise it is considerably disadvantageous. Moreover, since camera- based physiological sensing is a privacy sensitive application, it is preferred to store the data at local devices instead of streaming the video and physiological data cloud. The overhead from processing not affordable if we aim to make the system accessible to low-end mobile devices. 3) Many of these steps non- non-trivial to implement and optimize in and of themselves. This makes it harder to deploy real-time systems and to replicate the implementation on different platforms. For instance, implementing existing methods on Android, iOS, or in JavaScript requires significant amount of

---

Under review as a conference paper at ICLR 2022
1. Existing Leaminng Methoes
Meazurements
A Face Detection
B. RDI Segmentation
[C.ooorA Augmentation, RGB YUV
& Landmark Detection
High Dimensional Scale Spatial Temporal Map
Video Frames
Sparial
Averaging
Deep High Complexity
MMC
Bund Votuma Puba
Combinations
2. EfticeenPhyys
Efficient Spatial Temporal Modeling
Normaltzation Module
Temporal Module
Connected
Time
Conv2D
Diff
BatchNorm
W
OR
Bood
Transfomer Block 2D
Raw Framnes
Channel
Figure 1: High-level comparison between EfficientPhys and existing deep learning approaches for camera-based vitals measurement
effort. Some libraries, such facial landmark detection, not even available every platform Thus, the last mile engineering using the existing methods becomes especially challenging.
Ideally, video-based physiological measurement method would be able to run high frame rate even on mobile devices, simple to implement across different platforms, and achieve state-of-th titee-of-thhe the art performance. Addressing the aforementioned challenges would help achieve these properties. We propose a truly end-to-end network, EfficientPhys, for which the input is unprocessed video frames without requiring accurate face cropping (see Fig. D. Due recent interest in visual transformers, we propose both convolutional and visual transformer architecture and compare and contrast the performance of these two.
In summary, our key contributions are to: 1) propose two novel one-stop neural architectures, visual transformer and convolutional network, which do require any preprocessing steps, 2) evaluate the proposed methods on three popular benchmark datasets, 3) evaluate on on-device latency across both tfate-of-the-att machine learning based approaches as well as signal processing based techniques. To the best of our knowledge, this is the first paper that explores the visual transformer in camera-based physiological measurement and comparison with convolutional networks. This also the first paper exploring completely end-to-end neural architecture. Our code and project page is available at here and supplementary materials.
RELATED WORK
Camera-based Vital Measurement. There growing community studying the use cameras to sense physiological vitals signs (Wu et al. 2000 Takano & Ohta 2007 Verkruysse et al. 2008). Prior work established the fundamentals signal using signal source separation techniques (e.g., ICA) (Poh et al. 2010). Other methods derived separalio
how RGB images could be used to extract the pulse
these parameters from physically-ben basedd models to achieve elegant and fast demixing (e.g., Plane tithogonal-to-Skin (POS)) (Wang al., 2017). By calculating a projection plane orthogonal the skin-tone based on optical and physiological principles, the authors were able to achieve a stronger BVP signal-to-noiss -o-ise ratio (SNR).
Since the underlying relationship between the pulse and skin pixels is complex, deep convolutional neural networks have shown superior performance over the traditional source separation algorithms. DeepPhys (Chen & McDufi 2018b) was the first paper that demonstrated that a deep neural network outperforms all the traditional signal processing approaches. Liu et al. have also proposed an on- device efficient neural architecture called MITS-CAN for on namera-based physiological sensing, which leverages a tensor-shift module and 2D-convolutional operations to perform efficient spatial- temporal modeling (Liu et al. (2200)). More recently, an adversarial learning approach, called Dual- GAN, has also been studied to learn noiseanesistant mappings from video frames to pulse waveform and noise distributions (Lu et al. 202I) With two generaative-ddverariaa networks, they can promote
https: htt.://wwpgpg..cccoc////oooa/oae
2

---

Under review as a conference paper at ICLR 2022
Convolution based EfficientPhys
Rormalzation
Modul
M
Beichfrrm
Corry
Sal-essrcce
Bcat Pube
Nesk
Chandd
Transformer based EfficienPhyss
Stage
Stage
Stage
Stage
Mith
Block
Block
Sleck
Ubod
Raw Framis
Figure 2: We present two novel architectures to enable simple, fast and accurate camera-based vitals measurement: Convolution based EfficientPhys and Transformer based EfficientPhys. the number frames of video clip inputting to the network.
each adversarial network's 's representation and further improve the feature disentanglement between pulse and various noise sources.
However, DeepPhys and MTTS-CAN both require few preprocessing steps including calculating difference frames and performing image normalization. Dual-GAN has a even more complex preprocessing module called MSTMaps proposed by Niu et al. (020). The MSTMaps is multi- scale spatial temporal map by 1) fine-grained facial cropping, 2) landmarker extraction, 3) performing average pooling for every color channel and every ROI combination for each frame, 4) generating ROI combinations using all the detected ROI regions and landmarkers, 5) multiplications of each item all ROI combinations with six channel respectively. The final size of MSTMap (2" 1) x T x where 7 is the number of frames and n is the number of ROI regions. Such preprocessing module not only takes large memory size but also could significant introduces large computational burden to the entire pipeline. Moreover, stacking all of these extra procedures makes development and deployment much more difficult. Unlike these previous work, our the goal of proposed EfficientPhys is to create preprocessing -reee neural architecture that is simple use and deploy, efficient mobile devices, and accurate on settings with various noises.
Visual Transformers. Although convolutional neural networks have been widely studied and used in many computer vision applications, vision transformer started showing its superior performance on the task image classification task. By training larger datasets, vision transformer (VīT) attains excellent performance and can be used in downstream fine-tuning with fewer amount of data (Dosovitskiy al. 2020). More recently, the state-of-the-arr vision transformer, called Swin Transformer, was proposed to construct hierarchical feature maps and improve computational efficiency by using hierarchical representation and limiting self-attention computation non-overlapping local windows while allowing for cross-window connection (Liu et al. 2021b). However, transformer architectures have been merely studied in the field of camera-based vital measurement. The closest work using transformer to detect remote photoplethysmoogrppe (et al. 2021). However, this paper did not have evaluation on the proposed vision transformer in the task of heart rate estimation using any public datasets, which is considered as the gold-standard benchmark for the field of camera-based vital measurement. To our best knowledge, our proposed vision transformer is the first architecture in camera-buse based heart rate measurement with detailed evaluation on various public datasets.
(rPPG) for for attack/spoofing detection (Yu
3 METHOD
3.1 CONVOLUTION BASED EFFICIENTPHYS
To enable simple, fast and accurate real-time on-device camera-based vitals measurement, we propose a one-stop-solution architecture that takes raw video frames the input to the network and outputs a first-derivative PPG signal. The convolution based EfficientPhys is a one-branch network that contains custom normalization layer, self-attention module, tensor-shift module and 2D convolution modeling while making it simple to
operation to perform efficient and accurate deploy.
3

---

Under review as a conference paper at ICLR 2022
Normalization Module. Existing neural methods all require different levels of preprocessing before providing the visual representation to the network to learn the underlying relationship between skin pixels and cardiac pulse signal. For instance, The state-of- state-of-the-art the-art networks Dual-GAN (Lu et al. (021)) and CVD (Niu et al., 2020) proposed a hand-crafted spatial-temporal representations called S MaMss. These preprocessed representations are generated for each video frame and includes steps of detecting 81 facial landmark points, extracting a set of region of interest (ROI) combinations (2" - -1 where n the number of ROIs, n=6) using these landmarks, and averaging pixel values in both the RGB and YUV color spaces, multiplying the 63 ROI combinations with the six channels. These modules not only add significant computational burden (Table (shows that Dual-GAN's preprocessing module takes 275ms per frame) but also make the system more challenging to implement and deploy on real-world computing systems such as mobile devices.
One of the goals of EfficientPhys is to remove these preprocessing modules entirely and provide a one-stop solution. To achieve such simplicity and deployability, we propose a custom normalization module, which can perform motion modeling between every two consecutive frames and normaliza- tion to reduce the lighting and motion noise. More specifically, the proposed normalization module includes difference layer and batchnorm layer. The difference layer (e.g., torch.diff difff) computes the first forward difference along the temporal axis of the raw video frames, by subtracting every two adjacent frames. To provide optical basis in our work, equation [ illlstraaes the optical grounding of difference frame where (4) of every two consecutive frames where I(1) is the luminance intensity which is modulated by the specular reflection (t) and the diffuse reflection vt( as well as optical sensor's 's quantization noise U (t).
D=(1)= (T(1) (1) + va(()) + un ()) (T(1 1) (v.()))()())) va(l - 1)) + vn.( - 1)) (1)
(I(
However, difference frames could be dramatically different in scale and make it hard for the network to learn meaningful feature representations, especially when the signal of interest is hidden in subtle pixel changes along the temporal axis and noise artifacts can cause significantly larger relative changes. To address this, we add nomaliooraaiizaiion layer following by the difference layer. Adding a batchnorm layer provides two benefits: 1) it normalizes the difference frames to the same scale within the batch during training, 2) unlike fixed normalization in previous work (Chen & McDuff, 2018b; Liu et al. 2020a), batchnorm provides two learnable parameters ,3 and y for scaling (to different variance) and shifting (to a different mean) and two constant parameters which are the mcan and the standard deviation a. Through the learning process, the batchnorm layer can learn the best parameters for removing noise as the Equation 2 shows. Without a batchnorm layer, directly applying a difference layer means the frames appear "black"; because the subtle changes of skin pixels every two consecutive frames are relatively very small. On the other hand, adding a follow-up batchnorm layer will help it learn the normalization function to magnify the subtle changes of skin pixels substantially. The result is not simply a magnification of values but a normalization and magnification. Moreover, we also compare the output batchnorm layer to the hand-crafted normalized frame as shown in Fig The output of batchnorm layer contains more information and qualitative analysis suggests it should be better tool for skin segmentation after the learning process.
N M (1) (3. * (t) + 76) -DD.
OD
Self-Attention-Shiitte Network. To efficiently capture the rich spatial-temporal information, we propose a self-aatention-shiitee network (SASN). SASN is built on top of the previous state-of-the- art method for on-device spatial-tempont modeling in optical cardiac measurement tensor-shift convolutional attention network (TS-CAN) (Liu et al.) 2020b). TS-CAN has two convolutional branches, one of which takes a preprocessed difference frame representation and one of which takes a normalized appearance frame. The motion branch performs the main spatial-temporal modeling and estimation, and the appearance branch provides attention masks to guide the motion branch to better isolate the pixels of interest (e.g., skin pixels). However, argue that the attention masks do not have to be obtained through a separate appearance branch and they can be also learned with a single branch end-to-end network. As Fig. 2 iillustrates, our proposed custom normalization module discussed in the previous section then continues with two tensor-shifted convolutional operations. After the second and fourth tensor-shifted 2D convolutional layers, we add
iiitention-shittee network starts with the
4

---

Under review as a conference paper at ICLR 2022
Output of BatchNorm Layer
HandCrafted Normalized Frame
Raw Frame
Output of Diff
vs
Figure 3: Outputs of Diff and batchnorm layers and comparison with normalized frames generated via the hand-crafted process in prior work Chen & McDuff (2018b). The output from the diff layer is almost black because the difference in skin pixels of consecutive frames is very subtle.
a self-attention module respectively to help the network minimize the negative effects introduced by tensor shifting as well as motion and lighting noises. The self-attention layers are softmax attention layers with 1D convolutions followed by a sigmoid activation function. Then, normalization is applied remove the outlying values in the attention mask, and the final normalized attention mask is element-wise multiplied with the output from the tensor-shifted convolution. Equation summarize how our self-attention mechanism works where (x(.) denotes tensor shift operation, w denotes the 2D convolutional kernel followed by the tensor shift module, and w is the x 1 convolutional kernel for self attention.
(w)ts(N&(t)) b o H.W, (w + b%)
2 ow + +b,)
3.2 TRANSFORMER BASED PFFICIENTPHY'S
Efficient Spatial-Temporal Video Transformer: Due to the recent success of visual transformers for image and video understanding and the importance of attention mechanisms for this task (Chen & McDufff, 2018a Yu et al. 2019; Qi et al. 2020 Liu et al. 2020b), we also present a visual transformer version of EfficientPhys. For this task, we need a visual transformer learn both spatial and temporal representations. Several existing badco-based visual transformers are based on 3D-embedding tokens and input all the frames into 3D encoder and ppatial-temporal attention modules (Arnab et al., 2021: Liu et al., 2021d). However, the computational complexity makes that unfavourable for real-time efficient modeling on mobile devices. In the convolutional version we used tensor-shifted 2D convolutions which have been shown to achieve comparable performance as 3D convolutions (Liu et al. 2020b). Inspired by this, our proposed transformer based EfficientPhys based on 2D visual transformer, Swin transformer (Liu al. 2021c), but with added components that we will describe below.
Since the 2D Swin transformer is only able to learn spatial features that map RGB values to latent representations between a single frame and the target signal (pulse) and does not have ability to model temporal relationships beyond consecutive frames. One of the main contributions of the Swin transformer is the shifted window module which has linear computation complexity and allows cross-window connection by shifting the window partition and limiting self-attention computation to non-overlapping local windows. Inspired by the idea of shifting of spatial window partitions, we propose add tensor-shift module (TSM) (Lin et al. 2019) before every Swin transformer block to facilitate information exchange across the temporal axis. The TSM first splits the input tensor into three chunks, shifts the first chunk to the left by one place (advancing time by one frame) and shifts the second chunk to right by one place (delaying time by one frame). All the shifting operations are along temporal axis and performed before the tensor is fed into each transformer block as shown in Fig. By adding the TSM module to the Swin transformer, the new transformer architecture now has the ability to perform efficient spatial-temports modeling and attention by combining shifting window partitions spatially and shifting frames temporally. It is worth noting that TSM does not introduce any learnable parameters thus the proposed transformer architecture has the same number of parameters the original Swin transformer. Finally, enable truly end-to-end inference and
5

---

Under review as a conference paper at ICLR 2022
learning, we also add the same normalization module proposed in the convolution EfficientPhyys to this architecture.
In summary, the transformer- EfficientPhys is the first end-to-end transformer architecture for camera-based cardiac pulse measurement that leverages tensor-shift modules and window- window-partition shift modules perform efficient patialltemporal modeling and attention to learn the underlying physiological signal from skin pixels.
EXPERIMENTS
Training Data. To help create a robust and generalizable model for cross-datasct evaluation we use two datasets. The first is AFRL (Estepp et al., 2014), which includes 300 videos from 25 subjects (17 males and females). For each video, the raw resolution is 658x492 and the sampling rate of the synchronized pulse measurement is 30Hz. The dataset includes videos with range of head motions. Every participant was instructed to maintain stationary for the first two tasks, and then to perform head motions with increasing rotational velocity in the next four tasks (turning from left to right). Along with AFRL dataset, we also leverage a synthetic avatar video dataset introduced by McDuff et al. (220)) where each synthetic video is parameterized and generated with custom pulse signal, background, facial appearance, and motion. More specifically, the input pulse signal is used to augment skin color and the subsurface radius of skin pixels to mimic the effect of the blood volume pulse on the skin's 's appearance. Synthetic data such as this introduces greater diversity into the training set and has been shown to effectively help reduce disparities in performance by skin type.
Testing Data. We use three popular benchmark datasets to evaluate the accuracy of the proposed EfficientPhys. UBFC (Bobbia et al. 2019) is a dataset of 42 videos from 42 subjects, and the raw resolution of each video is 640x480 in uncompressed 8-bit RGB format. The sampling for synchronized pulse signal is 30 Hz. All of the tasks collected in UBFC are stationary. MMSE (Zhang et al. 2016) is dataset including 102 videos from 40 subjects, and the raw resolution of each video is at 1040x 1392. The ground-truth waveform for MMSE is blood pressure signal instead of blood-volume pulse signal, and the sampling rate is 25hz. It is worth noting that MMSE contains diverse distribution of skin types in Fitzpatrick scale (II=8, III111, IV=17, V+VI=4), PURE (Stricker) al., 20144) is a containing 60 videos from 10 subjects. The raw resolution of each video is 640x480, and the sampling rate of ground ground-truth pulse signal is 60 Hz. PURE includes a diverse set of motion tasks such as steady, talking, slow/fast translation between head movements and the camera plane, small/medium head rotation.
Implementation & Experiment Details. We implemented both convolution based and transformer based EfficientPhys in PyTorch (Paszke et al. 2019). We used an Adam optimizer to train both networks instead of Adam by introducing additional regularization to reduce the effects of over-fitting through weight decay (Loshchilov & Hutter (017)) The learning rate we used for Convolutional model was 0.001 while the one for transformer model was 0.0001. Based on empirical studies, used mean squared error(MSE) loss for training the transformer models and negative pearson loss (Tsou et al. 2020) for the convolutional model. We trained both models for five epochs. We implemented TS-CAN based the open-sourced code (Liu al. 2021a 2020b). To calculate the performance metrics, we first applied a band-pass filter to the signal with a cutoff frequency of 0.75 and 2.5Hz (45 beats/minute to 150 beats/miiutee.. We then followed DAN's evaluation scheme to run peak detection and FFT to get estimated heart rate on each video of UBFC and PURE datasets (Lu et al.) and MetaPhys's evaluation schema on MMSE (Liu et al., 202 We calculated three standard metrics for each video: mean absolute error (MAE), root mean squared error (RMSE) and Pearson correlation (p) in heart rate estimations and the corresponding ground -tuth heart rates from the blood volume pulse collected via contact oximeter sensor.
To explore the efficiency of different architectures on mobile devices, we also conducted experiments on a quad-core Cortex-A72 Raspberry Pi 4B to evaluate the model's performance on an edge device. We performed inference 10 times to get a reliable averaged on-device inference latency for EfficientPhys and TS-CAN. Due to the lack of open-source implementation from Dual-GAN, we were only able to find the implementation of STMaps which is the preprocessing module of Dual-GAN. Thus, only evaluated the on on-device latency for the preprocessing module in Dual-GAN We also evaluated the latency of POS, CHROM, and ICA as they are traditional signal processing methods and don't 't have a separate preprocessing module.

---

Under review as a conference paper at ICLR 2022
Table I: Cross-dataset heart rate evaluation on three public datasets: UBFC, MMSE and PURE (beats minute)
UBFC RMSEL
PURE RMSEL
MMSE RMSEL
Method
MAE1
P
MAEJ
p
MAEL
p
EfficientPhys-CC
1.14
1.81
0.99
1.33
5,99
0.97
2.91
5.43
0.92
EfficientPhys-T1
1.53
2.27
0.99
2.15
7.98
0.94
3.48
7.21
0.86
EfficientPhys
T2
3.07
4.78
0.96
7.71
9.95
0.85
3.51
6.98
0.88
TS-CAN(Liu al.
2020b
1.70
2.72
0.99
2.48
9.01
0.92
3.85
7.21
0.86
Dual-GANN(La et a
2021
0.74
1.02
0.99
N/A
N/A
N/A
N/A
N/A
N/A
PulseGAN Song ct al.
2021
2.09
4.42
0.99
N/A
N/A
N/A
N/A
N/A
N/A
POS( Wang et al.
2017
3.52
8.38
0.90
3.14
10.57
0.95
3.98
6.66
0.67
De Haan & Jeanne
2013
3.10
6.84
0.93
2.07
9.92
0.99
7.31
9.85
0.57
ICA et al.
2010
4.39
11.60
0.82
N/A
N/A
N/A
10.2
14.4
0.50
MAE Ment Absolute Enror HR estimation, RMSE
Root Mean Square Eroor
HR estimation, Pezzson Conrelation in HR extmmrion.
Table 2: On-Device Data Preprocessing Latency and Model Inference Latency Per Frame (ms)
Preprocessing
Model
Total
Method
(ms)
(ms) |
(ms) +
EfficientPhys-C
40
40
EfficientPhys-Tl
300
300
Phyis2entPhys-22
40
TS-CAN al.
2020b
60
63
Dual-GANdi et
202T
275
N/A
> 275
POS Wang et al.
2017
27
27
CHROM De Haan Jeanne
2013
28
Poh et al.
2010
31
31
Proprcosiing
lntenny Respberry per
5 RESULTS AND DISCUSSION
EfficientPhys vs. State-of-the-Art. In the Table 1 we present results from our proposed Efficient- Phys models and the current state-of-the-art neural and signal processing methods. The learning models are all trained on the same datasets and tested on separate dataset (c.g., UBFC) to test if the model can be generalize videos with a different facial appearance, background and lighting. To investigate how the depth of the network impacts the Transformer architecture, we created two version of Transformer- based Efficienthhys: T1 and T2. TI uses the same depth the Swin Transformer reported in Liu et al. 2021c) c) ([2. 2, 6, 21). Each number indicates the number of Swin Transformer blocks as illustrated in Fig. T2 has a much lighter architecture to enable real-time on-device inference which has depth of [2. 1]. EfficientPhys-CC denotes the Convolution bbased EfficientPhys as shown in the 2 EfficientPhys-C and EfficientPhys-Ti outperform almost all the existing argue that the margin is relatively small both Dual-GAN
method except Dual-GAN. However, and EfffcientPPys-C achieve a Pearson correlation of 0.99. EfficientPPys-C again surpasses all the state-of-the-art methods by 46% of MAE in PURE and 26% of MAE in MMSE. Unfortunately, Dual-GAN did not perform cross-dataset evaluation on other datasets. Due to the lack of open source implementation released models, we could not successfully replicate their complicated model architecture.
Computational Cost and On-Device Latency. Fig. and the Table summarize the computational cost of the existing neural methods. Again, due to the lack of open source implementation and complex algorithm design, we were not able to replicate every architecture benchmark its on-device latency. The results show that EfficientPhs-CC only takes 40ms to process single frame and it does not take any extra computational time to perform preprocessing. On the other hand, due to the complexity model architecture and extra time for calculating hand-crafted normalized raw and difference frames, TS- TS-CAN takes 63ms per frame. As mentioned earlier, Dual-GAN has a complicated preprocessing procedure for facial landmark detection, segmentation, color transformation and augmentation. We implemented this and benchmark the preprocessing module on our platform, and it takes 275ms per frame, which is already 7x than the entire computational time of EfficientPhys-C. The estimation
7

---

Under review as a conference paper at ICLR 2022
network in Dual- also includes 12 2D convolution operations, numerous 1D convolution operations. Thus, we believe it would add a significant amount of computational time on top of the 275ms preprocessing time per frame. The default Transformer-based EfficientPhys (TI) has unfavorable inference due to its deep architecture design and takes 300ms to process every single frame. After reducing the depth EfficientPhys-T2, it can achieve same inference time the EfficiemPPys-C.. However, EfficientPhys- has a much worse performance on all three benchmark datasets.
Convolution vs. Transformer in Camera- Based Vitals Measurement. Although visual transformers have begun to achieve state-of-the- art performance on some vision tasks, it is not the case in the task of video based vitals mea- surement. Based on the results shown in Table 1 Efficient-C outperforms both Efficient-T] by 25% MAE UBFC, 38% MAE in PURE, 16% of MAE in MMSE, while Efficient-C is more than 7x faster in terms of latency. When EfficientPhys Ef-
EfficientPhys-C 40ms/1.14bmp
Dual-GAN 285ms/0.74bmm
Latency of Preprocessing Module Only
Mean Absolute Error (BMP) in UBFC
1.5
TS-CAN
2.0
63ms/L.70bmp
EfficientPhys-11 300ms/3.07bmp
2.5
CHROM
28ms/3.10bmp
shrink the Transformer-base to a similar complexity as Convolution- ficientPhys, the accuracy performance signif-- icantly diminished. The eπrоrs from lightweight Transformer-basee EfficientPhys-T2 increased 106% of MAE in UBFC, 478% of MAE in PURE and 21% % of MAE in MMSE. These re- sults indicate shallow transformer architecture struggles to model subtle changes skin pixels in the video. These finding suggest two poten- tial insights. First, further optimizations will be necessary for transformers outperform, even relatively shallow, convolutional models in this domain, this is possibly especially true when there is not large amount of high-quality data available. As previous studies have shown (Dosovitskiy et al. 2020). Transformers usually require more pre-training samples to obtain state- of-the-art accuracy. Unfortunately, currently the amount of data in the field of camera-based vital measurement is limited compared to other visual tasks. We believe synthetic data is one way to help address this issue. Second, the good accuracy-efficieng trade-off for visual transformer might not be scaled to on-device architectures without further work. Since many on-device neural networks require significantly less amount of computing resources to perform real-time operations, scaling the Transformer architecture down is not ideal as experimental results of EfficientPhys-TT have shown.
3.0
EfficintPhys-T2
40ms/3.07bmp
3.5
POS
27ms/3.52bmp
ICA
4.5
31ms/4 31mm/4.39bmpp
100
200 Latency (ms)
150
250
300
Figure 4: Accuracy-Latency Trade-off in eight dif- ferent methods. Y-axis denotes the MAE error, and X-axis denotes the latency. The methods in the left-top corner have the best accuracy-latene latency Trade-off.
Simplifying Last-Mile ML Deployment. Numerous real-world applications are driven by novel machine learning algorithms. However, deploying these algorithms on different computing platforms has been extremely challenging for various reasons. One of these is researchers sometimes only pay attention to the accuracy of the model and ignore the complexity of the last-mile engineering efforts. In this paper, we address this important issue through our one-stop architecture which takes the unprocessed raw frames and directly outputs the desired signal. This elegant and simple design not only will reduce the burden of engineering required for cross-plas plassfoaa implementations, but also will help the research community to replicate and reproduce results.
Extensible to Other Signal. Finally, another potential upside of our end and-to-end design and the low latency, we envision EfficientPhys could be applied to various other video-bate based applications. Since the input of our model is raw frame, we believe EfficientPhys can be easily extended to other physiological signals and formats of data such as video-based blood pressure measurement and video understanding & recognition etc. On the other hand, most the baseline methods we compared (e.g.. Dual-GAN, PulseGAN) require many custom preprocessing operations for video-based measurement which are less useful in other applications.
8

---

Under review as a conference paper at ICLR 2022
6 BROADER IMPACTS AND ETHICS STATEMENT
Recent Information and Communications Technologies for Development (ICT4D) work reconceptu- alize technology development to empower the people serves. Machine learning based applications such as health interventions also tend to focus more on the development of the technology itself rather than the people and problems they address. Although many large-scale deep neural network models are trained on the data created by the public, they are often not freely available to the public. In this work we only used data that was collected under informed consent for the purposes of physiological analysis. We also make the trained models accessible and available to more diverse communities as described below.
During the development of EfficientPhys, intended that the innovations do not create larger disparities between different populations. By achieving the state-of-the-arr accuracy and efficiency as well as simple and elegant design, we believe EfficientPhys will help make camera-based vitals measurement more widely available to the medical research community and broader community in computing. We also believe that this technology can have partieular impact in low-resource settings there are greater barriers to access healthcare. We envision our proposed method could, with the appropriate clinical validation and regulatory approval, eventually be used in healthcare applications (e.g., real-ting vitals measurement in telehealth appointments). During the COVID-19 19 pandemic the need for such technology has been clearly highlighted. We contextualize our contributions within the scope of democratizing technology for social good and helping reduce health disparities with advanced AI technology. However, we are aware that machine learning systems are biased and can propagate inequalities. Before technology such as that presented in this paper is ready for deployment we need to make sure that that is not the case.
7 REPRODUCIBILITY STATEMENT
We described detailed experimental details in the section of 4. We also release our code in Github anonymous repo and provide a detailed project overview an anonymous website: https wttp:///wwo.ooeeeeeeeeeeeeeeeeee
CONCLUSION
In this paper, we present a novel method called EfficientPhys to enable simple, fast, accurate camera- based contactless vitals measuremnt. We achieved the state-of-ther the-art performance with using significant less computational power. With the simple and elegant one-stop design, EfficientPhys also help address the issue of last-time machine learning deployment and reduce health disparity.
REFERENCES
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. ViVIT: A Video Vision Transformer, arXiv:2 203.:203..10 22..2/03355500 15691 [cs], March 2021. URL http::/axx: arXiv: 2569.15691.
org/ hogg/bs/21000 2103. 15691
Serge Bobbia, Richard Macwan, Yannick Benezeth, Alamin Mansouri, and Julien Dubois. Unsu- pervised skin tissue segmentation for remote photoplethysmography. Pattern Recognition Letters, 124:82-90, 2019,
Weixuan Chen and Daniel McDuff. Deepphys: Video-based physiological measurement using convolutional attention networks. In Proceedings of the European Conference Computer Vision (ECCV), pp. 349-365, 2018a.
Weixuan Chen and Daniel McDuff. DeepPhys: Video-Bate Physiological Measurement Using Convolutional Attention Networks. arXiv:I805. 07888 [cs], August 2018b. URL http://
arxiv. oreiiioorg/bb... 1805. 07888
arXiv: 1805.07888.
Gerard De Haan and Vincent Jeanne. Robust pulse rate from chrominance- rppg. 1EEE Transactions Biomedical Engineering, 60(10):2878-2888 2013.

---

Under review as a conference paper at ICLR 2022
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Justin R. Estepp. Ethan Blackford, and Christopher M. Meier, Recovering pulse rate during motion artifact with multi-imager arry for non-contact imaging photoplethysmographyy In 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 1462-1460 1469, October 2014.
doi: 10.1 109/
ISSN: 1062-922X.
Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7093, 2019.
Xin Liu, Josh Fromm, Shwetak Patel, and Daniel McDuff. Multi-task temporal shift attention networks for on-device contactless vitals measurement. arXiv preprint arXiv:2006.03790, 2020a.
Xin Liu, Josh Fromm, Shwetak Patel, and Daniel McDuff. Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement. October 2020b.
Xin Liu, Ziheng Jiang, Josh Fromm, Xuhai Xu, Shwetak Patel, and Daniel McDuff. MetaPhys: Physiological Measurement. arXiv:2010.01773 [cs], March
Few-Shot Adaptation for 2021a. URL http: / arxiv. org abs /20002010../ 01773 arXiv: 2010.01773.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint 1xXi:22033114030 2021b.
Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, arXiv:2103, 100.10000 [es], March 2021c. URL httt:://arxiii org. hbtpg///bb///sa/a/ 14030 arXiv: 2103. 14030.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. arXiv preprint arXiv:2106. 13230, 2021d.
Hya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Hao Lu, Hu Han, and S Kevin Zhou. Dual-GAN: Joint BVP and Noise Modeling for Remote Physiological Measurement. pp. 10.
Hao Lu, Hu Han, and S Kevin Zhou. Dual-gam Joint bvp and noise modeling for remote physiological measurement. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12404-12413, 2021.
Daniel McDuff. Javier Hernandez, Erroll Wood, Xin Liu, and Tadas Baltrusaitis. Advancing Non- Contact Vital Sign Measurement using Synthetic Avatars. arXiv:2010. 12949 [cs], October 2020.
URL httt::tt/a//tiii org/ abs/2/00./2000.. 12949. arXiv: 2010.12949.
Xuesong Niu, Zitong Yu, Hu Han, Xiaobai Li, Shiguang Shan, and Guoying Zhao. Video-based re- mote physiological measurement via cross-verified feature disentangling. In European Conference Computer Vision, pp. 295-310. Springer, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32: 8026-8037, 2019.
Ming ZZer Poh, Daniel J McDuff, and Rosalind W Picard. Advancements in noncontact, multiparam- eter physiological measurements using a webcam. IEEE transactions on biomedical engineering. 58(1):7-11 2010.
10

---

Under review as a conference paper at ICLR 2022
Hua Qi, Qing Guo, Felix Juefei-Xu, Xiaofei Xie, Lei Ma, Wei Feng, Yang Liu, and Jianjun Zhao. Deeprhythm: Exposing deepfakes with attentional visual heartbeat rhythms. In Proceedings of the 28th ACM International Conference Multimedia, pp. 2918-4327, 2020.
Dangdang Shao, Yuting Yang, Chenbin Liu, Francis Tsow, Hui Yu, and Nongjian Tao. Noncontact monitoring breathing pattern, exhalation flow rate and pulse transit time. IEEE Transactions on Biomedical Engineering. 61(11):2760-2760 2014.
Rencheng Song, Huan Chen, Juan Cheng, Chang Li, Yu Liu, and Xun Chen. Pulsegan: Learning to generate realistic pulse waveforms in remote photoplethysmography. IEEE Journal of Biomedical and Health Informatics, 25(5):1373-133 2021.
Ronny Stricker, Steffen Müller, and Horst-Michael Gross. Non-contact video-based pulse rate measurement mobile service robot. In The 23rd IEEE International Symposium Robot and Human Interactive Communication, pp. 1056-1062. IEEE, 2014.
Chihiro Takano and Yuji Ohta. Heart rate measurement based on a time-lapse image. Medical engineering & physics, 29(8):853 -57,, 2007.
Kenji Takazawa, Nobuhiro Tanaka, Masami Fujita, Osamu Matsuoka, Tokuyu Saiki, Masaru Aikawa, Sinobu Tamura, and Chiharu Ibukiyama. Assessment of Vasoactive Agents and Vascular Aging by the Second Derivative of Photoplethysmogram Waveform. Hypertension, 32(2):36--370 August 1998. doi: 1161/01 HYP 32. w2..2.2.... 365
URL /ttp:ttt/////t http::/wwww.aha.ouuaaaaaaaaaaa wttp:///aaaa..oooooo//oooo/////
Yun- Yun-Yun Tsou, Yi-An Lee, Chiou-Ting Hsu, and Shang-Hung Chang. Siamese-rppg network: Remote photoplcthysmographyy signal estimation from face videos. Proceedings of the 35th annual ACM symposium on applied computing, pp. 2066-2077 2020.
Wim Verkruysse, Lars O Svaasand, and Stuart Nelson. Remote plethysmographic imaging using ambient light. Optics express, 16(26):11444-11455 2008
Wenjin Wang, Albertus C. den Brinker, Sander Stuijk, and Gerard de Haan. Algorithmic Principles of Remote PPG. IEEE Transactions Biomedical Engineering. 64(7):1479-1491, July 2017. ISSN 1558-2531. doi: 10. 109/TBMM. 2609922.. Conference Name: IEEE Transactions on Biomedical Engineering.
Ting Wu. Vladimir Blazek, and Hans Juergen Schmitt. Photoplethysmography imaging: new noninvasive and noncontact method for mapping of the dermal perfusion changes. In Optical Techniques and Instrumentation for the Measurement of Blood Composition, Structure, and Dynamics, volume 4163, pp. 62-70. International Society for Optics and Photonics, 2000.
Zitong Yu, Xiaobai Li, and Guoying Zhao. Remote Photoplethysmograph Signal Mcasurement from Facial Videos Using Spatio-Temporaa Networks. arXiv: 001999950000244 [cs], July 2019. URL http://arxivw org abs 905 02419
arXiv: 1905.02419.
Zitong Yu, Xiaobai Li, Pichao Wang, and Guoying Zhao. Transrppg: Remote photoplethysmography transformer for 3d mask face presentation attack detection. IEEE Signal Processing Letters, 2021.
Zheng Zhang. Jeff M Girard, Yue Wu, Xing Zhang, Peng Liu, Umur Ciftei, Shaun Canavan, Michael Reale, Andy Horowitz, Huiyuan Yang, al. Multimodal spontaneous emotion corpus for human behavior analysis. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 3438-3446, 2016.
A
APPENDIX
Project and Code Link:
11


Reviews:
[{"review_id": "y1n2pf9aen0", "reviewer": "Reviewer_S4mf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The novelty of the approach resides in its aim to efficiency, mainly obtained by removing the preprocessing phase. The specific convolutional version leverages a spatial-temporal attention based on a parameter-less tensor shift, with a further gain in efficiency. \n- The outline of the paper is clear and it is generally well written, with some minor exception. Both the convolution and transformer approaches are described in detail. \n- The comparison shows that convolutional version of EfficientPhys outperforms other state-of-the-art approaches. Interestingly, it also shows that the this variant is more efficient than the transformer-based one since it obtains a greater accuracy if the models are constrained to the same size. It would be interesting to explore how much general is this assessment about the comparison of the two architecture.\n- references seem accurate, comprehensive and up to date.\n- code is released as public github repository.\n\nWeaknesses:\n- The choice to call with the same name (EfficientPhys) two different approaches is a bit confusing. Models are indeed three: EfficientPhys-C (convolution), EfficientPhys-T1 (transformer-regular), EfficientPhys-T2 (transformer-shrinked). They share the same normalization block and arguably a similar output block, but with a different core. They also perform in a very different way: only '-C' outperforms other approaches in terms of MAE, RMSE, ro (except DualGAN), and is the Pareto optimum for MAE/latency; '-T1' is good in terms of MAE, RMSE, ro, but not in latency; '-T2' in contrast show interesting latency but not so good accuracy. It is difficult to consider these models as unique, and results can not be claimed for EfficientPhys but for EfficientPhys-C.\n- the lack of a complete comparison with the best other approach, DualGAN, is disappointing. As explained by the authors, this is due to the lack of results of DualGAN on the public datasets PURE, MMSE, and cannot be totally addressed to them. \nNonetheless, this diminishes the claimed results since DualGAN outperforms EfficientPhys models in UBFC dataset.\n- Self-Attention-Shifted Network is described by eq. 3 which is too verbose and somehow obscure, and needs to be better explained as it describes a core component of the model. It fails to give an intution of how the module works.\n- It is not explained at all which is the final task (and consequently modules on top of architecture) of both models: probably regression?\n- Fig. 2: the normalization module seems different in the two versions, but reading the text it seems the very same. Figures are great value for giving an intuition of how a system works, but a standardization of the pictograms is needed.\n- Fig. 4 is a bit confusing in the 0/50 latency range, 2.5/4.0 MAE: the chosen symbols overlap.\n- minor problems about the text: \n-- pag. 4, after eq. 1: 'To address this, we add a batch-normalization layer followed by the difference layer' should be: 'To address this, we add a batch-normalization layer following the difference layer'\n-- pag. 5, second paragraph of section 3.2: the first sentence, 'Since the 2D Swin transfromer...', is not correct and needs to be rephrased\n-- pag. 9, last two sentences of section 6: 'However, we are aware...': they are quite involuted and the meaning is not clear.\n", "summary_of_the_paper": "The paper introduces two novel neural models to address the task of camera-based physiological measurement: the first model leverages convolutional layers and a spatio-temporal attention module; the second model is based on Vision Transformer.\nThey are described as variants of the same network called EfficientPhys, and both remove the need of specific preprocessing steps on input data by introducing a normalization module.\nThe resulting lightweight architecture makes both variants easy to be implemented in multiple devices such as mobiles, and to be applied to the elaboration of different physiological signals.\nEfficientPhys is tested on three public datasets with state-of-the-art approaches obtaining good results in both accuracy and latency.\n", "summary_of_the_review": "The paper is an interesting contribution in the task of camera-based physiological measurement.\nTwo neural models are presented with two different approaches: convolution, and vision transformer.\nThey are alternative version of the same network called EfficientPhys, and represent valuable examples of efficient deep learning: they show a lightweight pipeline having removed the data preprocessing, and a spatio-temporal attention build on the parameterless tensor shift (for the convolution version). Nonetheless, they obtain good performance in terms of accuracy (MAE, RMSE, ro) and latency, with the convolutional model in the Pareto frontier.\nThe inherent lightness of the architecture makes the models portable to multiple devices platforms, facilitating the access to healthcare in low resources scenarios.\nFor these reasons the paper is to be accepted, with minor changes mainly to improve the readability.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lBz5PAA0riz", "reviewer": "Reviewer_kqwX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Let me summarize the strengths of the paper first.\n+ The presented approach is simple and easy to implement.\n+ There is an exploration of different designs of the backbone networks.\n+ The performance of the presented method is strong.\n\nThe weaknesses of the paper are as follows.\n- The writing of the paper needs improvements. There are many grammatical errors. For example, in the abstract, \"state-the-art-performance\" and \"Prior research have\" -> \"Prior research has.\" The authors are recommended to have a few revisions on the paper.\n- I do not think the presented architectures are technically novel. They are built with existing backbones. Also having an end-to-end network in deep learning applications is straightforward.\n- There are no ablation studies on different designs in the presented model. \n- The results on UBFC by the proposed method are not very competitive.\n- I do not get the point of presenting a model with transformers. This model apparently performs worse than the one with convolutions in accuracy and speed.", "summary_of_the_paper": "In this work, the authors present simple and efficient models for camera-based vital measurement. The proposed approach does not rely on any pre-processing steps but uses end-to-end neural networks. The authors present two types of neural networks, one based on convolutional neural networks and one based on transformers. The experiments show the approach archives state-of-the-art performance on three public datasets with fast inference speed. ", "summary_of_the_review": "In summary, I feel the work lacks sufficient technical contribution in terms of novelty and empirical motivation. The experimental results are good but somehow expected. The writing of the paper needs improvements.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bv5AQK2GNBb", "reviewer": "Reviewer_anoh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n\n1.  The application is important and it will certainly be easier to deploy models which have fewer requirements around input data preprocessing - so I think the motivation for the paper is good and I appreciate the emphasis on trying to make these systems useful in the real world.\n2. I believe this is the first successful application of a transformer architecture for remote ppg estimation. The approach seems clean. The introduction of a tensor-shift module before Swin transformer blocks seems to be a neat way to get some temporal propagation of information. The CNN based model also appears to be simpler than the previous work it builds on, by removing the need for an input appearance branch and associated compute by virtue of adding a self attention layer. It is interesting to me that using batch-normalized difference frames at input appears to be sufficient for extraction of the ppg signal.\n3. The cross-dataset experiment is a good way to evaluate generalization, because different datasets are known to exhibit different kinds of biases.\n4. Releasing the code will be beneficial to others in the community.\n\n# Weaknesses\n\n1. The estimation task in the paper is concerned with photoplethsymography/heart rate. Although the authors hint to the use of the model for other estimation tasks (e.g. blood pressure) in future, it is not actually assessed. Since \"vitals\" and \"physiological measurement\" encompass more than heart rate estimation (e.g. respiratory rate, blood pressure, temperature), I believe this work should be described as a technique for remote PPG / BVP / heart rate monitoring (and not the more generic \"vitals\" or \"physiological measurement\") unless experiments can be added which demonstrate otherwise. To avoid misleading readers, I think this ought to be made clearer in both the title and the abstract.\n\n2. The paper clarity could be improved in places and sometimes appears to lift from the work of MTTS-CAN while losing a few important details, e.g.\n- it is unclear what the value of Eq 1 is to understand this work - none of these terms are ever mentioned again. Perhaps the motivation underlying this equation could be better explained through simple English or a figure?\n- Figure 2 does not label the dense layer; \"volumne\" typo; Temporal Shift Module should ideally be referenced? Is the self-attention mask illustrated here an actual output from the EfficienctPhys-C model?\n- Fig 4 which is similar to a figure from the MTTS-CAN paper but here the circle radius is unused? Could the circles be resized? Y axis should be BPM?\n- Eqn. 3 is not very clear to understand, without being familiar with the “Attention on Temporal Shift” section of the TS-CAN paper. Please add further explanation with reference to the past work to explain how it improves. Please also choose a different term than `ts` to denote tensor-shift, as the `t` of `ts` may be confused with time.\n\n3. While I agree that pre-processing can make deployment complicated, I don't think this issue has been sufficiently explored to be able to call this a technical claim of the paper. It should also be pointed out that certain preprocessing steps such as face tracking are increasingly readily available, even on lightweight devices. The fact that the proposed model doesn't require face tracking is likely to be because it is (1) implicitly doing it, thanks to the training data and/or (2) makes assumptions such as the face being roughly centered, taking up a certain proportion of the image, and the background remaining static throughout. I think the paper would benefit from pointing out or exploring these things. Taken to the extreme, if the input data is 10MPx, and the subject's face takes up 0.1 MPx, then a lightweight face detection module could potentially save a lot of compute/memory that these proposed methods would not -- so even though this model does not require preprocessing, it does rely on assumptions about how the input data is captured, which should be clarified.\n\n4. At low error rates on these three test datasets, errors caused by one or two subjects or video clips can make a big difference between \"SOTA\" or not. The datasets themselves are also known to contain erratic ground truth. Was any filtering carried out to remove erroneous samples from the evaluation? Do EfficientPhys-C and EfficientPhys-T1 tend to make similar errors on the test datasets?\n\n5. I understand from Table 2 that  “The learning models are all trained on the same datasets”, but this probably only applies to the methods which have all columns populated? For example, Dual-GAN’s numbers for UBFC are taken from Table 5 of their paper, which is based on training on PURE and testing on UBFC. In addition Dual-GAN reports rho as 0.997 which has been rounded to 0.99 here. I think it should be made clear that this is not a fair comparison, and it is unclear which method actually benefits from this unfairness. It is no fault of the authors that Dual-GAN’s method is hard to reproduce, but it would be good to be more transparent here even if it makes the narrative a bit murkier.\n\n6. The paper highlights the benefits of batch normalization, but other than a qualitative example in Fig 3, does not show any experiment results to highlight its benefits. Moreover, the benefits of batch normalization may depend on the batch size used in training, but I could not find any details about this in the paper. Please could these details be added to the paper?\n\n7. TS-CAN seems to be the main benchmark for this work. I understand EfficientPhys-C differs in several ways (removal of appearance branch, additional of bnorm, addition of self-attention); it would be valuable to see the effects of these various changes - do they rely on each other to get the performance benefit?\n\n8. I understand EfficientPhys-T1 and T2 differ by the number of blocks, [2,2,6,2] -> [2,1]. Were any other alterations attempted to reduce the compute requirements, such as varying patch sizes, or number of heads, etc?\n\n9. Reproducibility: it is great that training code for this model can be shared and that the evaluation is on public datasets. However, is the synthetic dataset available for other researchers to be able to reproduce this work? If the synthetic data cannot be shared prior to the conference, the paper would benefit by providing alternative reproducible experiments - e.g. leave-one-out cross dataset training on public datasets, otherwise it will not be possible for others to build on this work.\n\n10. More characteristics about the synthetic data should be recapped in Section 4, as is done for the other datasets (e.g. resolution used, diversity etc.). What proportion of the final training dataset was real vs synthetic? Were any experiments undertaken to measure the variation of EfficientPhys models’ performances vs amount of synthetic data (the paper mentions “We believe synthetic data is one way to help address this issue [of limited availability of high-quality training data]” - was this measured in practice)?\n\n# Minor points\n1. Fig 1 is misleading. (1) makes it sounds like all existing deep learning methods have this preprocessing approach, which is false. For example the Yu et al reference \"Remote Photoplethysmograph Signal Measurement from Facial Videos Using Spatio-Temporal Networks\" (which in the References section should be updated to BMVC 2019) only assumes preprocessing with a (very lightweight and commonly available) Viola-Jones face detector. It should be made clear that (1) refers to specific approaches, not *all* existing approaches. \n\n2. The use of bolding in Table 1 is abused a bit, because EfficientPhys-C is bolded even when it is not the best result for the given metric. Stating that \"we argue that the margin is relatively small as both Dual-GAN and EfficientPhys-C achieve a Pearson correlation of 0.99\" does not justify bolding the values for EfficientPhys-C for MAE and RMSE, because this is a subjective assessment and the reader should be allowed to make their own subjective assessment. \n\n3. Fig 2: N does not appear to be defined anywhere in the paper? \n\n4. What are the limitations of this method? Based on Table 1, it appears that access to training data *might* be one which limits the transformer approach. Could another be that the difference layer only operates across contiguous frames? If the differences across contiguous frames were sufficiently small that they were drowned out by quantization noise, there might still be detectable differences across longer time periods. In such a case it might help to aggregate infomation across bigger frame gaps before throwing it away. It would be nice to see some discussion of the limitations of the approach here?\n\n5. The paper mentions “second forward difference” but isn’t this a 1st order difference? i.e. torch.diff(n=1) rather than torch.diff(n=2)? second forward difference means x_{t+2} - 2x_{t+1} + x_t?\n\n6. “the task of heart rate estimation... is considered as the gold-standard benchmark for the field of camera-based vital measurement” -- what is meant by gold-standard benchmark? It seems to imply that other vital sign measurements are not as important? The last few sentences of Visual Transformers part of Sec 2 have a few other typos/grammar errors.\n\n7. It is also interesting to me what the performance gap is when inverting the train and test datasets -- what is the gap when testing on synthetic data? The nice thing there is that synthetic data ground truth should contain no errors, and it would also allow more careful analysis of any algorithmic bias.\n\n8. The first paragraph on p4 “Normalization Module” describes related work, so probably belongs in a different section.\n\n9. Table 1. Are the results here based on one training run per model? It is useful to understand the variation in performance per model when starting from multiple random seeds.\n", "summary_of_the_paper": "This work proposes and contrasts two new approaches to estimate a photoplethysmography signal from facial videos using a supervised CNN- and transformer-based model.\nBoth models use a batch-normalized difference image input; the CNN model introduces the use of self-attention masks as opposed to more commonly used appearance-based attention masks, and the transformer model is based on the SwinTransformer model with tensor-shift modules to propagate information temporally.\nThe approaches are designed towards real-world deployment, by eliminating any pre-processing requirements and reducing on-device latency.\nThe models are evaluated in a cross-dataset setting, training on a combined real and synthetic dataset, and testing on three separate held-out real datasets. This evaluation shows that the proposed approaches, in particular the convolutional approach, perform well.\nThe accuracy vs latency trade-off is also assessed, showing that the approach can be made to run relatively efficiently on a Raspberry Pi Model 4B.\n", "summary_of_the_review": "I thought this paper was generally well written and I appreciate the overall goal of it, which is to make camera-based measurement of certain vital signs more easily accessible and therefore useful. I also appreciate the effort to apply and adapt a new technique (vision transformers) to the problem, as well as the benefits of adjustments to TS-CAN to improve its performance. While the results on the cross-dataset experiment look strong, I find the technical contributions and analysis to be a little light for the level I would expect of ICLR and I would have preferred to have seen more careful study of a single method to understand its benefits. For example, if focused on EfficientPhys-C, what are the effects of the changes to the model vs TS-CAN? Or if focused on EfficientPhys-T, what other architectural alterations could be explored to improve latency? I think that the approach introduced in the paper is good and that with some additional care towards framing, explanation and analysis, this can be a nice paper and contribution to the physiological monitoring literature.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Yc56LAFuyAJ", "reviewer": "Reviewer_B8NS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper shows superior results on multiple public datasets that are moderately difficult. One dataset include darker skin tones which are known to be more difficult for rPPG, and another dataset includes some head motion. The algorithm proposed are fast and can be deployed on mobile platform, which will help improve access to the technology in general, even though, for the topic at hand, accessibility is not the main focus here.\n\nThe machine learning aspect of this paper seems solid. The training and testing sets are clearly separated. All the basic information about the datasets that are important to understanding how ML is applied is clearly laid out in the experiment section.\n\nMy main concerns for this paper is around evaluation. Several aspects of evaluation needs to be improved:\n  - The test datasets are all relatively small. The biggest dataset only have 100 videos. Furthermore, the diversity and difficulty of the datasets are unclear. UBFC contains stationary videos, MMSE has only 4 subjects of the darkest skin types, and PURE only have at most moderate head motion. I recommend the author to perform evaluation on larger, more challenging datasets such as MAHNOB-HCI, and VIPL-HR, which contains many more videos (2k in the latter, and several hundered in the former).\n  - Some of the comparison in Table 1 is misleading, and I would love some additional clarification from the authors. The result for Dual-GAN seems to be copied from Table 5 of the original paper, where the model is trained on PURE, and test on UBFC. Since the training data is different from the current method (AFRL), the comparison is not direct. It's possible that Dual-GAN will do even better if trained on AFRL, or the other way around, where the proposed method could be even better if trained on PURE. I recommend the author to actually retrain comparisons on the exact same dataset, so that we can be sure that the proposed method actually improve on the state of the art.\n\n  In this particular case, Dual-GAN actually have better numbers than the proposed method. I would like to get some clarification on how the rest of the comparison is done. This concerns also apply to the efficiency evaluation. If the competing method have not been properly optimize, its runtime can suffer significantly. How has the authors optimized competing method's implementation onto their Raspberry Pi platform?\n\nSecondary to evaluation, it is unclear how swin transformer and tensor shifting actually help authors avoid preprocessing steps required by other methods. These are the components that have been used in previous work, and they still require some preprocessing in order to achieve good performance. One possible explanation is the choice of testing dataset that are relatively easy and contain limited amount of motion. I would love to see a more apple-to-apple comparison with previous methods (using the same dataset, well-tuned model) before I am convinced that the proposed method actually works.\n\nRelatedly, the authors claim to be the first end-to-end network, but I could find their custom normalization and diffing to be a sort of preprocessing, in the same spirit as some of the previous work (DeepPhys by Chen and McDuff).\n\nLastly, the method section contains many unjustified technical decision, and also some inaccurate statements. I would love to see the author's clarification on these:\nPage 4:\n“which can perform motion modeling between every two consecutive frames and normalization to reduce the lighting and motion noise.” – This claim that the normalization module is able to reduce lighting and motion noise hasn’t been shown in this paper.\n\nEquation 1: camera noise has more components than just quantization. Shot noise, read noise, are two other important sources of noise that was not considered here. There are also processing functions (sensor response can be non-linear), that were not discussed or taken into consideration. While these could be approximated away in most cases, the authors need to be precise about their treatment of these effects.\n\n“the batchnorm layer can learn the best parameters for removing noise as the Equation 2 shows.” Assuming that noise between sample is IID, why is it desirable to normalize across batch?\n\nlast paragraph: “we argue that the attention masks do not have to be obtained through a separate appearance branch and they can be also learned with a single branch end-to-end network” What is the author’s justification for this?\n\nPage 5:\nFigure 3: It’s unclear what difference color pattern means. These could easily be an artifact of color scale used in plotting. Also, if the concern is the difference frame will be close to zero, wouldn’t normalize it to zero mean, unit SD be enough?\n\nAdditional suggestions:\n\nAdditional results by groups could shed more light on what are the weaknesses and strengths of the proposed method. For example, how does the model perform in each sky type group, each motion amount group, etc.\n", "summary_of_the_paper": "This paper propose an end-to-end learning-based method to extract heart rate from facial videos. The authors proposed two networks--a convolutional based, and transformer-based. Both network utilize tensor shifting to encode temporal dimension. The Conv-based one utilize self-attention to guide the network towards important region of the image, while the transformer-based one uses swin transformers. The authors implemented their network in rasberry pi and show that their method is more accurate than the comparison on three relatively small public datasets, while being 33% faster in processing time due to lack of preprocessing.", "summary_of_the_review": "At face value, this paper produce superior HR detection result, and more efficient algorithm. However, at closer inspection, a more direct comparison, and evaluation on a more challenging dataset are still needed to validate the effectiveness of the algorithm. In my opinion, the contribution is still not enough for a top-tier venue like ICLR.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a model for heart rate estimation from video. Unlike previous approaches, the method does not perform pre-processing of the video (face detection, cropping, etc). The empirical results are good - on par with state of the art or sometimes better.

Most reviewers are negative even after considering the authors' responses, but there is no full consensus. Some mentioned pros of the paper are that it's practically useful to have a model without pre-processing steps and that overall the technical side of the paper seems sound. Some cons are that the technical novelty is limited, the empirical evaluation is somewhat limited, and appropriate ablation studies are missing.

Overall, I recommend rejection at this point. While the paper goes in a promising direction, it is an application paper and as such it would be expected to have a more solid experimental evaluation - ideally on multiple tasks (as the paper title might suggest) and on more datasets as mentioned by the reviewers and the commenters. Moreover, an ablation study (and/or other analysis) of the proposed model would be crucial to let the readers know what components of the model actually contribute to its performance.
Decision: Reject