SlowGuess/NeurIPS_2023_OCR huggingface dataset:
With regard to the training of neural networks, our toolbox provides flexibility to parameterize which portion of the data is used for training, validation, or testing. For instance, we can use the first  $80\%$  of UBFC-rPPG for training, the last  $20\%$  of UBFC-rPPG for validation and then use the entire PURE dataset for testing. Moreover, the distinct parameters (e.g., dropout rate) of each neural network can be defined in the config file.
For post-processing and evaluation, there are several standard post-processing steps that are typically employed to improve model predictions. A 2nd-order Butterworth filter (cut-off frequencies of 0.75 and  $2.5\mathrm{Hz}$ ) is applied to filter the predicted PPG waveform. The choice of filtering parameters can have a significant impact on downstream results such as heart rate errors. A Fast Fourier Transform or a peak detection algorithm is then applied to the filtered signal to calculate the heart rate. In this toolbox, we support five metrics for video-level heart rate estimations: mean absolute error (MAE), root mean squared error (RMSE), mean absolute percentage error (MAPE), signal-noise ratio (SNR), and Pearson Correlation  $(\rho)$ , along with a calculation of standard error for a better understanding of the accuracy of the aforementioned metrics. We also give users the option to visualize Bland-Altman plots as a part of evaluation. Finer details on the supported metrics (F), metric results not reported in the main paper (G), and Bland-Altman plots (J.3) appear in the respective appendices. For better reproducibility, we also provide pre-trained models in our Github repository to allow researchers to perform model inference. The detailed definition of each config parameter is also provided in the GitHub repository.
3.4 Benchmarking
To show that the implementations of the baseline methods are functioning as expected and to provide benchmark results for consumers of the toolbox to reference and reproduce, we performed a set of baseline experiments using three commonly used video rPPG datasets for training: SCAMPS [24], UBFC-rPPG [22] and PURE [23] and tested on four datasets including UBFC-rPPG [22], PURE [23], UBFC-Phys [27], and MMPD [25]. Except where noted in the GitHub repository, neural models utilized a training batch size of 4, 30 epochs, and an inference batch size of 4 for all experiments. Due to the multi-institution team behind this toolbox, different kinds of GPUs were utilized to produce benchmark results. Tables 2 and 4 were produced on a machine using a single NVIDIA RTX A4500 GPU. Tables 3 and 5 were produced on a machine using a single NVIDIA GeForce 2080 Ti GPU. As illustrated in Table 2, we show MAE and MAPE computed between the video-level heart rate estimations and gold standard measurements. Additional cross-dataset experiment results and metric results can be found in Appendix-G and F.
4 Additional Features
4.1 Weakly Supervised Training
Supervised rPPG training requires high fidelity synchronous PPG waveform labels. However not all datasets contain such high quality labels. In these cases we offer the option to train on synchronous PPG "pseudo" labels derived through a signal processing methodology as described by [41]. These labels are produced through POS-generated [31] PPG waveforms, which are then bandpass filtered around the normal heart-rate frequencies (cut-off frequencies of 0.70 and  $3.0\mathrm{Hz}$ ), and finally amplitude normalized using a Hilbert-signal envelope. The tight filtering and envelope normalization results in a strong periodic proxy signal, but at the cost of limited signal morphology.
For instance, in the BP4D+ dataset [26], the cardiac ground truth is represented by a blood pressure waveform. Although this waveform exhibits the same periodicity as the PPG signal, it has a phase shift that adversely affects model training. A figure that illustrates sample pseudo labels derived for BP4D+ [26] videos plotted against the ground truth blood pressure waveform as shown in Figure 3. Table 3 presents results for supervised methods, trained on BP4D+ [26] pseudo labels. We extend this feature to all of the supported datasets.
Table 3: Training with Pseudo Labels. For the supervised methods we show results training with the (entire) BP4D+ [26] dataset, using POS [31] derived pseudo training labels.
Training Set	BP4D+[26] with POS Pseudo Labels
Testing Set	UBFC-rPPG [22]	PURE [23]
MAE↓	MAPE↓	ρ↑	MAE↓	MAPE↓	ρ↑
Supervised
TS-CAN [6]	4.69	4.51	0.78	1.29	1.60	0.97
PhysNet(Normalized) [5]	1.78	1.92	0.96	3.69	7.35	0.88
DeepPhys [35]	2.74	2.81	0.93	2.47	2.49	0.89
EfficientPhys-C [36]	2.43	2.52	0.90	3.59	3.27	0.80
MAE = Mean Absolute Error in HR estimation (Beats/Min), MAPE = Mean Percentage Error (\%),  $\rho =$  Pearson Correlation in HR estimation.
4.2 Motion Augmented Training
The usage of synthetic data in the training of machine learning models for medical applications is becoming a key tool that warrants further research [42]. In addition to providing support for the fully synthetic dataset SCAMPS [24], we provide provide support for synthetic, motion-augmented versions of the UBFC-rPPG [22], PURE [23], SCAMPS [24], and UBFC-PHYS [27] datasets for further exploration toward the use of synthetic data for training rPPG models. The synthetic, motion-augmented datasets are generated using an open-source motion augmentation pipeline targeted for increasing motion diversity in rPPG videos [43]. We present cross-dataset results using a motion-augmented version of the UBFC-rPPG [22] dataset in Table 4. We also provide tools that leverage OpenFace [44] for extracting, visualizing, and analyzing motion in rPPG video datasets. Further details regarding these tools are shared in our GitHub repository.
Table 4: Training with Motion-Augmented Data. We demonstrate results training on a motion-augmented (MA) version of the UBFC-rPPG [22] dataset generated using an open-source motion augmentation pipeline [43] and testing on the unaugmented version of the PURE [23] dataset.
Training Set
Testing Set	PURE [23]	MAUBFC-rPPG [22]	MMPD [25]
MAE↓	MAPE↓	ρ↑	MAE↓	MAPE↓	ρ↑	MAE↓	MAPE↓	ρ↑
TS-CAN [6]	1.07	1.20	0.97	5.03	6.36	0.75	12.59	13.77	0.23
PhysNet (Normalized) [5]	17.03	32.37	0.38	5.51	7.50	0.68	10.67	13.99	0.33
DeepPhys [35]	1.15	1.40	0.97	4.95	6.26	0.75	12.71	13.70	0.21
EfficientPhys-C [36]	2.59	2.67	0.88	4.80	6.10	0.79	13.39	14.50	0.14
MAE = Mean Absolute Error in HR estimation (Beats/Min), MAPE = Mean Percentage Error (%),  $\rho =$  Pearson Correlation in HR estimation.
4.3 Extending the rPPG-Toolbox for Physiological Multitasking
While this toolbox is primarily targeted towards rPPG model training and evaluation, it can be easily extended to support multi-tasking of physiological signals. As an example, we implement BigSmall [41], an architecture that multi-tasks PPG, respiration, and facial action. Similar to [41] we present 3-fold cross-validation results across the action unit (AU) subset of BP4D+ [26] (the portion of the dataset with AU labels), and use the same subject folds and hyperparameters as implemented in the original publication. These results can be found in Table 5. Note, that like [41], facial action metrics are calculated across 12 common AUs (AU #s 1, 2, 4, 6, 7, 10, 12, 14, 15, 17, 23, 24). Additional details of training and evaluation could be found in Appendix-I.
Time (Seconds)
Figure 3: Generated Pseudo Labels. Samples of POS [31] generated PPG pseudo labels plotted against ground truth blood pressure waveforms from BP4D+ [26].
Table 5: Multitasking Results. For the BigSmall [41] method we show results for multi-tasking PPG, respiration, and action unit classification; training with the BP4D+ [26] (AU subset) dataset, using POS [31] derived pseudo training PPG labels.
Training Set	BP4D+ [26]
Testing Set	BP4D+ [26]
Task	rPPG	Respiration	Facial Action
MAE↓	MAPE↓	ρ↑	MAE↓	MAPE↓	ρ↑	F1↑	Prec. ↑	Acc. ↑
BigSmall [41]	3.23	3.51	0.83	5.19	26.28	0.14	42.82	39.85	65.73
MAE = Mean Absolute Error in HR estimation (Beats/Min), MAPE = Mean Percentage Error  $(\%)$ ,  $\rho$  = Pearson Correlation in HR estimation, F1 = average F1 across 12 action units, Prec. = average precision across 12 action units, Acc. = average accuracy across 12 action units.
4.4 Training, Evaluation and Analysis Features
In this toolbox, we have incorporated a diverse set of training and evaluation functionalities. These include: 1) data pre-processing visualization tools, enabling users to inspect and understand their data before feeding it into the model; 2) comprehensive tracking and visualization of key training metrics such as training loss, validation loss, and learning rate, facilitating a thorough monitoring of the model's learning progress; 3) the implementation of Bland-Altman plots, providing a robust method for assessing agreement between two ground-truth HR and predicted HR; 4) advanced motion analysis capabilities. For an in-depth exploration of these features, we direct the reader to Appendix-J and our Github page, where detailed descriptions and usage examples are provided.
We acknowledge there are many limitations in our current toolbox and plan to continue improve it in the future. In the ensuing phases of our research, we envision a collaborative approach, working in concert with the wider scientific community, to enhance the efficacy and capabilities of the rPPG-Toolbox. The current limitations include 1) this toolbox does not support all of the latest neural architectures and diverse datasets; 2) it does not support unsupervised and self-supervised learning paradigms; 3) it does not support applications beyond of heart rate calculation such as heart rate variability, blood pressure,  $\mathrm{SpO}_2$ , and other important physiological measures.
6 Broader Impacts
Camera sensing has advantages and benefits with the potential to make important cardiac measurement more accessible and comfortable. One of the motivating use-cases for rPPG is turning everyday devices equipped with cameras into scalable health sensors. However, pervasive measurement can also feel intrusive. We are releasing the rPPG toolbox with a Responsible AI License [45] that
restricts negative and unintended uses of the toolbox. We also acknowledge the presence of several potential negative concerns and impacts, which are described as follows.
Privacy Concerns: Camera-based physiological sensing offers a revolutionary way to extract physiological signals from video recordings, enabling a myriad of applications, from remote patient monitoring to daily fitness tracking. However, these advancements come with significant privacy concerns. First and foremost, the very nature of remote sensing allows for the collection of personal data without direct physical interaction or, in some cases, knowledge of the individual being monitored. This can potentially enable unauthorized entities to capture sensitive physiological data covertly. Furthermore, as these systems become more widespread, there is a risk that everyday places such as shopping malls, public transport, and even workplaces might employ rPPG systems, leading to widespread passive data collection. Such extensive monitoring can lead to privacy invasions, where individuals are constantly under physiological surveillance without their explicit consent.
Potential Negative Impact: Beyond individual privacy, there are broader societal implications of widespread contactless camera-based physiological sensing. There's the potential for the creation of a pervasive surveillance state where citizens are continuously monitored, not just for their actions but also for their physiological responses. Such monitoring can lead to "physiological profiling," where individuals are judged, categorized, or even discriminated against based on their bodily responses. For instance, elevated heart rates or other physiological markers might be misinterpreted as signs of nervousness, guilt, or deceit, potentially affecting decision-making in areas such as law enforcement, job interviews, or public services. Moreover, a continuous emphasis on physiological metrics might foster an environment of physiological conformism, where people feel pressured to exhibit 'normal' physiological signs even if they aren't feeling well or are under duress.
Potential Ethical Concerns: The ethical implications of rPPG are multi-faceted. Firstly, there is the concern of informed consent. As technology becomes more integrated into our environments, it becomes challenging to ensure that individuals are aware of, and have agreed to, the collection of their physiological data. Moreover, the accuracy and reliability of rPPG systems can vary depending on factors like skin tone, lighting conditions, and other external factors. This introduces the risk of systematic biases, where certain groups might be inaccurately assessed or marginalized due to technological limitations or inherent biases in the algorithms. Ethical concerns also arise from potential misuse. For example, businesses might use rPPG data to gauge consumer reactions to products or advertisements, leading to manipulative strategies that target individual vulnerabilities. In extreme cases, authoritarian regimes might use such technologies to monitor citizens for signs of dissent or unrest. As with any potent tool, the ethical application of rPPG requires careful consideration of its potential for both benevolent and malevolent use.
Research relies on the sharing of ideas, this not only allows methods to be verified, saving time and resources, but also allows researchers to more effectively build upon existing work. Without these resources and open-sourced code bases, fair evaluation and comparison of methods is difficult, creates needless repetitions, and wastes resources. We present an end-to-end and comprehensive toolbox, called rPPG-Toolbox, containing code for pre-processing multiple public datasets, implementations of supervised machine learning (including training pipeline) and unsupervised methods, and post-processing and evaluation tools.
This project is supported by a Google PhD Fellowship for Xin Liu and a research grant from Cisco for the University of Washington as well as a career start-up funding grant from the Department of Computer Science at UNC Chapel Hill. This research is also supported by Tsinghua University Initiative Scientific Research Program, Beijing Natural Science Foundation, and the Natural Science Foundation of China (NSFC).
[66] Theodore Curran, Xin Liu, Daniel McDuff, Shwetak Patel, and Eugene Yang. Camera-based remote photoplethysmography to measure heart rate and blood pressure in ambulatory patients with cardiovascular disease: Preliminary analysis. Journal of the American College of Cardiology, 81(8_Supplement):2301-2301, 2023.
[67] Ying-Ying Zheng, Yi-Tong Ma, Jin-Ying Zhang, and Xiang Xie. Covid-19 and the cardiovascular system. Nature Reviews Cardiology, 17(5):259–260, 2020.
[68] Mauricio Villarroel, Sitthichok Chaichulee, João Jorge, Sara Davis, Gabrielle Green, Carlos Arteta, Andrew Zisserman, Kenny McCormick, Peter Watkinson, and Lionel Tarassenko. Noncontact physiological monitoring of preterm infants in the neonatal intensive care unit. npj Digital Medicine, 2(1):1-18, 2019.
Our appendices contain the following additional details and results:
- In Section B, we provide details regarding the background of rPPG and an overview of existing methods.
- In Section C, we provide an overview of potential applications out of rPPG technologies.
- In Section D, we provide additional results about PhysNet.
- In Section E, we provide an overview of rPPG network recommendations.
- In Section F we provide details toward metrics supported by our toolbox. We also provide additional metric results in Section G that were not included in the main paper due to space constraints.
- Section H briefly details which subjects we utilized for exclusion, or conversely sub-selection, in each task when dealing with the UBFC-Phys [27] dataset. We also briefly describe video filtering criteria available via the toolbox and useful for subject sub-selection.
- Additional details related to training and evaluation for physiological multitasking is shared in Section I.
- Section J briefly describes additional features included in the toolbox. These features, including pre-processed data visualization, loss and learning visualization, Bland-Altman plots, and motion analysis, are further detailed with exemplar usage in the rPPG-Toolbox's GitHub repo: https://github.com/ubicomplab/rPPG-Toolbox
B Background and Existing Research in rPPG
B.1 Background
Figure 4: The principles behind camera-based physiological sensing. The volumetric changes of blood under the surface of the skin cause changes in light absorption and reflection, which is the source of PPG signal.
Remote PPG (rPPG) measurement involves the development of computational methods for extracting physiological parameters (e.g., pulse rate, respiration rate, blood oxygenation, blood pressure) based on light reflected from, or transmitted through, the human body. Essentially these methods use pixel information to quantify changes in visible light, or other electromagnetic radiation (e.g., infrared or thermal), that are modulated by blood flow in the periphery of the skin (see Fig.4). This reflected radiation is also affected by body motions and absorption characteristics of the skin [46, 47, 48, 49, 50, 51, 31]. In this article, we will focus primarily on the use of visible light, due to the ubiquitous nature of RGB cameras.
As visible light penetrates between 4 to  $5\mathrm{mm}$  below the skin's surface, it is modulated by the volume of oxygenated and deoxygenated hemoglobin enabling the measurement of the peripheral blood
volume pulse (BVP). The frequency channels offered by multiband (e.g., RGB) cameras enable the composition of blood, including the oxygen saturation to be measured. In addition, these pixels are affected by the motion as a person breaths in and out and by the mechanical effects of the heart beating, enabling the measurement of breathing signals and the ballistocardiogram (BCG). Analyzing the morphology of these signals, and combining them together, offers the possibility of measuring correlates of blood pressure.
B.2 Optical Principals
The Lambert-Beer law (LBL) [50, 51, 35] and Shafer's dichromatic reflection model (DRM) [31] are two models which provide a framework for capturing the effects of an imager, lighting, body motions and physiological processes on recorded pixel intensities. Given the optical characteristics of oxygenated and deoxygenated blood, we also have priors on the wavelengths of light that contain the strongest or weakest pulsatile information. This prior knowledge is important for measuring physiological parameters accurately. Most computational methods are built upon this grounding.
B.3 Algorithms
Many computational approaches for recovering physiological signals from videos have similar steps. The first step typically involves localizing a region of interest within each video frame. In a large majority of cases the face or head are the region of interest and therefore facial detection and/or landmark detection are used. However, in other cases skin segmentation might be preferred. Aggregating pixels spatially is a subsequent step that has been used to help to reduce noise from camera quantization errors. The operation can be performed by downsampling an image [35] or simply averaging all pixel values with a region of interest [49]. Many cameras capture frames from more than one frequency band (e.g., RGB) that provide complementary measurements to capture different properties of the light reflected from the body. This information can be used in two ways: 1) for understanding the composition of the blood (e.g., oxygen saturation), 2) improving the signal-to-noise ratio of the recovered blood volume pulse. Typically, computational methods leverage multiple bands and learn a linear or non-linear signal decomposition to estimate the pulse waveform. This manipulation of the color channel signals can be grounded in the optical properties of the skin [31, 30] or learned in a data-driven manner given a specialized learning criteria or loss function.
More recently supervised machine learning has become the most popular approach. Specifically, deep learning and convolutional neural networks provide the current state-of-the-art results. These methods present the opportunity for more "end-to-end" learning and researchers have gradually tried to replace handcrafted processing steps with learnable components. Since the relationship between underlying physiological signal and skin pixels in a video is complicated, deep neural networks have shown superior performance on modeling such non-linear relationship compared to traditional source separation methods [35, 5, 52, 53, 6, 54, 55, 56, 57, 34]. Moreover, due to the flexibility of neural network, researchers have also explored neural based methods for real-time on-device inference [6, 36, 58, 59, 60, 61], self-supervised learning [14, 53, 62, 15, 13, 16], domain generalization [11, 63, 64] and estimating new vital measurement such as blood pressure [65, 66].
C Potential Applications of rPPG
The SARS-CoV-2 (COVID-19) pandemic has accelerated the pace of change in healthcare services. In particular, how healthcare services are delivered around the world has needed to be rethought in the presence of new risks to patients and providers and restrictions on travel. The virus has been linked to the increased risk of cardiopulmonary (heart and lung related) illness with symptoms such as respiratory distress syndrome, myocarditis, and the associated chronic damage to the cardiovascular system. Experts suggest that particular attention should be given to cardiovascular protection during treatment of COVID-19 [67]. While measurement is not the sole solution to these problems, they have acutely highlighted the need for scalable and accurate physiological monitoring. Ubiquitous or pervasive health sensing technology could help patients conduct daily screenings, monitor the effects of medication on their symptoms, and help clinicians make more informed and accurate decisions.
The potential advantages that video-based contactless measurement offers have helped to draw a significant amount of attention to the field in recent years. Contact biomedical sensors (e.g.,
electrocardiograms, pulse oximeters) are the standard used for clinical screening and at-home measurement. However, these devices are usually bulky and are still not ubiquitously available, especially in low-resource settings. On the other hand, non-contact camera-based physiological sensing presents a new opportunity for highly scalable and low-cost physiological monitoring through ordinary cameras (e.g., webcams or smartphone cameras) [29]. Besides the convenience and potential scalability, this technology could also reduce the risk of infection for vulnerable patients and discomfort caused by obtrusive leads and electrodes [68]. Finally, we believe there are two specifically compelling advantages of cameras over contact sensors. The first, is that they can capture multi-modal signals, including but not limited to, the activity of the subject, their appearance, facial expressions and gestures, motor control and context. One reason this helps is that the physiological measurements can be interpreted in context. For example, if someone appears in pain, an elevated heart rate can be interpreted differently than without in pain. Secondly, cameras are spatial sensors allowing for the measurement of signals from multiple parts of the body to be measured concomitantly, presenting greater opportunities for characterizing vascular parameters such as pulse transit time.