SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We would also argue that camera-based physiological sensing could be an influential technology in telehealth. Current telehealth procedures are mainly telephone or video-based communication services where patients see their physician or healthcare provider via Cisco Webex, Zoom or Microsoft Teams. Performing clinical visits at home increases the efficiency of clinical visits and helps people who live in remote locations. There is still a debate over whether high-quality care can be delivered over telehealth platforms. One of notable issues with current telehealth systems is that there is no way for physicians to assess patient's physiological states. The development of accurate and efficient non-contact camera-based physiological sensing technology would provide remote physicians access to the physiological data to make more informed clinical decisions.
D Investigation of PhysNet
Table 6: PhysNet Ablation Results. Investigation of variants of PhysNet models on the PURE [23] and UBFC-rPPG [22] datasets obtained using the rPPG toolbox.
Method	Train Set	Test Set
PURE [23]	UBFC-rPPG [22]
MAE↓	MAPE↓	MAE↓	MAPE↓
PHYSNET-RAW [5]	UBFC-RPPG	19.25	33.75	N/A	N/A
PURE	N/A	N/A	11.17	11.64
SCAMPS	18.40	31.74	10.57	11.04
PHYSNET-DIFFNORM [6]	UBFC-RPPG	8.06	13.67	N/A	N/A
PURE	N/A	N/A	0.98	1.12
SCAMPS	13.30	20.10	5.40	5.42
PHYSNET-RAW-TUNED [35]	UBFC-RPPG	4.80	8.46	N/A	N/A
PURE	N/A	N/A	1.99	1.86
SCAMPS	14.35	23.00	10.33	10.71
Metrics explained: MAE = Mean Absolute Error in HR estimation (Beats/Min), MAPE = Mean Percentage Error (\%).
Based on the rPPG community's help and suggestions, we found that there are two important implementation details missed in the original PhysNet paper: 1) the raw input range has to be set to 0-1; 2) adding normalization to the model output. However, with these changes and the raw input format, PhysNet is not able to achieve results that are close to SOTA in UBFC and PURE. As shown in Table 6's PhysNet-Raw-Tuned, we further fine-tuned the training parameters (changed learning rate from 0.009 to 0.09) and post-processing bandpass filter parameters (set bandpass filtering parameters to [0.5,2.5], detrend value to 200). However, we don't recommend using these network specified parameters as it is not a fair comparison across different network architectures. This toolbox aims to provide a standardized training regime across all the networks and datasets. All in all, we recommend using DiffNorm frames as the input to PhysNet to make training easier to converge.
We also notice that results while training on SCMAPS and testing on PURE are not good. It is worth noting that SCAMPS is a synthetic dataset and can easily cause overfitting with a complex network architecture. Unlike other 2D-CNN based networks, PhysNet is a 3D-CNN based network which is easier to overfit simple datasets.
E Network Recommendation
In the dynamic realm of rPPG research, architectural choices critically influence the balance between efficiency and accuracy. This section elucidates our findings and offers recommendations on architectures, factoring both computational aspects and performance nuances.
- Mobile and Computational Efficiency: For scenarios demanding computational frugality, such as mobile or edge deployments, 2D-CNN based architectures are recommended. They gracefully balance performance and computational overhead, ensuring expeditious responses in resource-limited environments.
- High-Performance Scenarios: 3D-CNN or Transformer-based networks, while resource-intensive, offer superior performance. These architectures are apt for applications with lenient resource constraints where the pinnacle of accuracy is sought.
- Performance Saturation: Notably, a saturation trend was observed across several network architectures. This insinuates diminishing performance returns with increased complexity or depth, emphasizing the need to pragmatically select architectures based on actual task necessities.
A salient lesson from our research emphasizes the indispensability of diverse dataset incorporation. The rPPG community grapples with challenges such as: 1) motion-infused videos, introducing substantial rPPG signal noise; 2) videos under different lighting conditions (e.g., natural light, LED, incandescent), affecting visual cues critical for models; 3) featuring darker-skinned individuals, historically underrepresented, leading to potential model biases. To bolster robustness and cater to diverse real-world scenarios, it's imperative to amalgamate videos encapsulating the aforementioned conditions in training sets. Such holistic training paradigms ensure the model's aptitude in handling a vast spectrum of real-world challenges.
F Metric Details
F.1 rPPG Metrics
We present explanations of metrics supported by our toolbox below.
Mean Absolute Error (MAE): For predicted signal rate  $R_{p}$ , ground truth signal rate  $R_{g}$ , and for  $N$  instances:
$$
M A E = \frac {1}{N} \sum_ {n = 1} ^ {N} | R _ {g} - R _ {p} |
$$
Root Mean Square Error (RMSE): For predicted signal rate  $R_{p}$ , ground truth signal rate  $R_{g}$ , and for  $N$  instances:
$$
R M S E = \sqrt {\frac {1}{N} \sum_ {n = 1} ^ {N} (R _ {g} - R _ {p}) ^ {2}}
$$
Mean Absolute Percentage Error (MAPE): For predicted signal rate  $R_{p}$ , ground truth signal rate  $R_{g}$ , and for  $N$  instances:
$$
M A P E = \frac {1}{N} \sum_ {n = 1} ^ {N} \left| \frac {R _ {g} - R _ {p}}{R _ {g}} \right|
$$
Pearson Correlation  $(\rho)$ : For predicted signal rate  $R_{p}$ , ground truth signal rate  $R_{g}$ , and for  $N$  instances, and  $\overline{R}$  the average of  $R$  for  $N$  samples:
$$
\rho = \frac {\sum_ {n = 1} ^ {N} \left(R _ {g . n} - \overline {{R _ {g}}}\right) \left(R _ {p . n} - \overline {{R _ {p}}}\right)}{\sqrt {\left(\sum_ {n = 1} ^ {N} R _ {g . n} - \overline {{R _ {g}}}\right) ^ {2} \left(\sum_ {n = 1} ^ {N} R _ {p . n} - \overline {{R _ {p}}}\right) ^ {2}}}
$$
Signal-to-Noise Ratio (SNR): As in [30], we calculate the Signal-to-Noise Ratio (SNR) for a predicted signal as the ratio between the area under the curve of the power spectrum around the first and second harmonic of the ground truth heart rate frequency and the area under the curve of the rest of the power spectrum. This is mathematically represented as follows:
$$
S N R = 1 0 l o g _ {1 0} \left(\frac {\sum_ {4 5} ^ {1 5 0} (U _ {t} (f) S (f)) ^ {2}}{\sum_ {4 5} ^ {1 5 0} ((1 - U _ {t} (f)) S (f)) ^ {2}}\right)
$$
Where  $S$  is the power spectrum of the estimated rPPG signal.  $U_{t}(f)$  is equal to 1 around the first and second harmonics of the ground truth rPPG signal, while being 0 elsewhere in the power spectrum. In the context of the rPPG-Toolbox, only the power spectrum between  $0.75\mathrm{Hz}$  and  $2.5\mathrm{Hz}$ , or 45 beats/min and 150 beats/min, is considered. We report the mean of the SNR values calculated per video or test sample, such that:
$$
M S N R = \frac {1}{N} \sum_ {n = 1} ^ {N} S N R
$$
Standard Error  $(\pm \mathrm{SE})$ : The standard error is a measure of the statistical accuracy of an estimate, such as the mean, and is equal to the standard deviation of the theoretical distribution of a large population of such estimates. The standard error takes into account the number of samples utilized in measurement, which is especially useful in the case of remote PPG datasets where the number of test samples can vary significantly from dataset to dataset. For all metrics except for the Pearson correlation  $(\rho)$ , we calculate the standard error as:
$$
S E = \frac {\sigma}{\sqrt {n}}
$$
Where  $\sigma$  is the standard deviation and  $n$  is the number of samples. For the Pearson correlation  $(\rho)$ , the standard error is calculated as:
$$
S E _ {\rho} = \sqrt {\frac {1 - r ^ {2}}{n - 2}}
$$
Where  $r$  is the correlation coefficient and  $n$  is the number of samples. Similar to how a standard deviation is reported, we report standard error as  $\pm SE$ .
F.2 Additional Multitask Metrics
We present explanations of additional metrics added to evaluate the BigSmall [41] model in order to exemplify how this toolbox can be extended to support physiological multitasking.
Evaluated Action Units (AU): Similar to [41], and other AU literature, facial action metrics are calculated for the following 12 commonly used AUs: AU01, AU02, AU04, AU06, AU07, AU10, AU12, AU14, AU15, AU17, AU23, AU24.
F1: The harmonic mean of recall and precision. For true positive count  $TP$ , false positive count  $FP$ , and false negative count  $FN$ .
$$
F 1 = 1 0 0 * \frac {2 T P}{2 T P + F P + F N}
$$
Precision (Prec.): For true positive count  $TP$ , and false positive count  $FP$ :
$$
P r e c i s i o n = 1 0 0 * \frac {T P}{T P + F P}
$$
Accuracy (Acc.) For true positive count  $TP$ , true negative count  $TN$ , false positive count  $FP$ , and false negative count  $FN$ :
$$
A c c u r a c y = 1 0 0 * \frac {T P + T N}{T P + T N + F P + F N}
$$
We reiterate results provided in the main paper and present additional results including the RMSE, SNR, Pearson correlation, and the corresponding standard errors. Note that there may be minor differences between results in the following tables and the main paper, as they were generated on a different machine using the latest version of the rPPG-Toolbox.
Table 7: Benchmark Results. Performance on the PURE [23] and UBFC-rPPG [22] datasets obtained using the rPPG toolbox. For the supervised methods we show cross-dataset training results using the UBFC-rPPG, PURE, and SCAMPS datasets.
Test Set
PURE [23]
Method	Train Set	MAE↓	RMSE↓	MAPE↓	ρ↑	SNR↑
UNSUPERVISED	GREEN [28]	N/A	10.09 ± 2.81	23.85 ± 217.81	10.28 ± 2.33	0.34 ± 0.12	-2.66 ± 1.43
ICA [29]	N/A	4.77 ± 2.08	16.07 ± 153.84	4.47 ± 1.65	0.72 ± 0.09	5.24 ± 1.77
CHROM [30]	N/A	5.77 ± 1.79	14.93 ± 81.53	11.52 ± 3.75	0.81 ± 0.08	4.58 ± 0.85
LGI [33]	N/A	4.61 ± 1.91	15.38 ± 134.14	4.96 ± 1.72	0.77 ± 0.08	4.50 ± 1.21
PBV [32]	N/A	3.92 ± 1.61	12.99 ± 123.60	4.84 ± 1.49	0.84 ± 0.07	2.30 ± 1.31
POS [31]	N/A	3.67 ± 1.46	11.82 ± 66.87	7.25 ± 3.03	0.88 ± 0.06	6.87 ± 0.95
SUPERVISED	TS-CAN [6]	UBFC-RPPG	3.69 ± 1.74	13.8 ± 113.84	3.39 ± 1.44	0.82 ± 0.08	5.26 ± 1.11
SCAMPS	4.66 ± 1.68	13.69 ± 92.53	5.83 ± 2.03	0.82 ± 0.08	0.95 ± 1.04
PHYSNET [5]	UBFC-RPPG	8.06 ± 2.34	19.71 ± 129.34	13.67 ± 4.04	0.61 ± 0.11	6.68 ± 1.16
SCAMPS	13.30 ± 2.00	20.30 ± 94.85	20.01 ± 2.97	0.51 ± 0.11	-8.73 ± 0.78
PHYSFORMER [34]	UBFC-RPPG	12.92 ± 2.69	24.36 ± 132.24	23.92 ± 5.22	0.47 ± 0.12	2.16 ± 1.05
SCAMPS	26.58 ± 2.14	31.24 ± 133.33	42.79 ± 4.06	0.12 ± 0.13	-12.56 ± 0.53
DEEPPHYS [35]	UBFC-RPPG	5.54 ± 2.30	18.51 ± 173.09	5.32 ± 1.90	0.66 ± 0.10	4.40 ± 1.32
SCAMPS	3.96 ± 1.67	13.44 ± 98.86	4.25 ± 1.60	0.83 ± 0.07	5.07 ± 1.15
EFF.PHYS-C [36]	UBFC-RPPG	5.47 ± 2.10	17.04 ± 143.80	5.40 ± 1.76	0.71 ± 0.09	4.09 ± 1.16
SCAMPS	10.24 ± 2.48	21.65 ± 173.96	11.70 ± 2.28	0.46 ± 0.12	-5.49 ± 1.05
Test Set
UBFC-rPPG [22]
Method	Train Set	MAE↓	RMSE↓	MAPE↓	ρ↑	SNR↑
UNSUPERVISED	GREEN [28]	N/A	19.73 ± 3.75	31.00 ± 235.38	18.72 ± 3.33	0.37 ± 0.15	-11.18 ± 1.63
ICA [29]	N/A	16.00 ± 3.09	25.65 ± 163.58	15.35 ± 2.77	0.44 ± 0.14	-9.91 ± 1.78
CHROM [30]	N/A	4.06 ± 1.21	8.83 ± 33.93	3.84 ± 1.10	0.89 ± 0.07	-2.96 ± 1.18
LGI [33]	N/A	15.80 ± 3.67	28.55 ± 236.17	14.70 ± 3.20	0.36 ± 0.15	-8.15 ± 1.41
PBV [32]	N/A	15.90 ± 3.25	26.40 ± 199.71	15.17 ± 2.91	0.48 ± 0.14	-9.16 ± 1.35
POS [31]	N/A	4.08 ± 1.01	7.72 ± 21.87	3.93 ± 0.91	0.92 ± 0.06	-2.39 ± 1.14
SUPERVISED	TS-CAN [6]	PURE	1.30 ± 0.40	2.87 ± 3.05	1.50 ± 0.47	0.99 ± 0.02	1.49 ± 1.13
SCAMPS	3.62 ± 0.91	6.92 ± 18.30	3.53 ± 0.84	0.93 ± 0.06	-3.91 ± 0.98
PHYSNET [5]	PURE	0.98 ± 0.35	2.48 ± 2.55	1.12 ± 0.42	0.99 ± 0.02	1.09 ± 1.15
SCAMPS	5.40 ± 1.46	10.89 ± 48.53	5.43 ± 1.38	0.82 ± 0.09	-4.97 ± 1.03
PHYSFORMER [34]	PURE	1.44 ± 0.54	3.77 ± 7.93	1.66 ± 0.62	0.98 ± 0.03	0.18 ± 1.12
SCAMPS	4.56 ± 1.46	10.48 ± 68.96	5.18 ± 1.93	0.81 ± 0.09	-6.34 ± 0.80
DEEPPHYS [35]	PURE	1.21 ± 0.41	2.90 ± 3.75	1.42 ± 0.49	0.99 ± 0.02	1.74 ± 1.16
SCAMPS	3.10 ± 1.44	9.81 ± 74.70	3.08 ± 1.32	0.87 ± 0.08	-0.79 ± 1.22
EFF.PHYS-C [36]	PURE	2.07 ± 0.92	6.32 ± 32.01	2.10 ± 0.87	0.94 ± 0.05	-0.12 ± 1.20
SCAMPS	12.64 ± 3.15	23.99 ± 182.44	11.26 ± 2.67	0.34 ± 0.15	-9.36 ± 1.05
MAE = Mean Absolute Error in HR estimation (Beats/Min), RMSE = Root Mean Square Error in HR estimation (Beats/Min), MAPE = Mean Percentage Error (%),  $\rho =$  Pearson Correlation in HR estimation, SNR = Signal-to-Noise Ratio (dB) when comparing predicted spectrum to ground truth spectrum.
Table 9: Training with Motion-Augmented Data. We demonstrate results training on a motion-augmented (MA) version of the UBFC-rPPG [22] dataset generated using an open-source motion augmentation pipeline [43] and testing on the unaugmented versions of the PURE [23] dataset, UBFC-Phys [27], and MMPD [25] datasets.
Training Set
Testing Set
Metric (± Std. Err.)	MAUBFC-rPPG [22]
PURE [23]
MAE↓	RMSE ↓	MAPE↓	ρ↑	SNR ↑
Supervised
TS-CAN [6]	1.07 ± 0.75	5.89 ± 33.75	1.20 ± 0.83	0.97 ± 0.03	8.86 ± 0.95
PhysNet (Normalized) [5]	17.03 ± 2.97	28.50 ± 149.16	32.37 ± 5.82	0.38 ± 0.12	7.27 ± 0.88
DeepPhys [35]	1.15 ± 0.76	5.95 ± 33.75	1.40 ± 0.85	0.97 ± 0.03	9.94 ± 1.00
EfficientPhys-C [36]	2.59 ± 1.43	11.29 ± 96.01	2.67 ± 1.27	0.88 ± 0.06	6.75 ± 1.12
Training Set
Testing Set
Metric (± Std. Err.)	MAUBFC-rPPG [22]
UBFC-Phys [27]
MAE↓	RMSE↓	MAPE↓	ρ↑	SNR ↑
Supervised
TS-CAN [6]
PhysNet (Normalized) [5]
DeepPhys [35]
EfficientPhys-C [36]	5.03 ± 0.67	8.39 ± 18.26	6.36 ± 0.90	0.75 ± 0.07	-1.15 ± 0.81
5.51 ± 0.88	0.44 ± 37.65	7.50 ± 1.32	0.68 ± 0.07	-0.57 ± 1.08
4.95 ± 0.67	8.37 ± 21.53	6.26 ± 0.90	0.75 ± 0.07	-0.78 ± 0.85
4.80 ± 0.58	7.52 ± 15.02	6.10 ± 0.79	0.79 ± 0.06	-0.87 ± 0.86
Training Set
Testing Set
Metric (± Std. Err.)	MAUBFC-rPPG [22]
MMPD [25]
MAE↓	RMSE↓	MAPE↓	ρ↑	SNR ↑
Supervised
TS-CAN [6]	12.59 ± 0.62	20.23 ± 31.27	13.77 ± 0.62	0.23 ± 0.04	-9.19 ± 0.29
PhysNet (Normalized) [5]	10.68 ± 0.49	16.56 ± 19.72	14.01 ± 0.72	0.32 ± 0.04	-9.28 ± 0.21
DeepPhys [35]	12.71 ± 0.65	21.04 ± 35.40	13.70 ± 0.64	0.21 ± 0.04	-8.85 ± 0.31
EfficientPhys-C [36]	13.42 ± 0.66	21.64 ± 35.46	14.52 ± 0.65	0.14 ± 0.04	-9.20 ± 0.31
MAE = Mean Absolute Error in HR estimation (Beats/Min), RMSE = Root Mean Square Error in HR estimation (Beats/Min), MAPE = Mean Percentage Error (\%),  $\rho =$  Pearson Correlation in HR estimation, SNR = Signal-to-Noise Ratio (dB) when comparing predicted spectrum to ground truth spectrum.
Table 10: Training with Pseudo Labels. For the supervised methods we show results training with the (entire) BP4D+ [26] dataset, using POS [31] derived pseudo training labels.
Training Set	BP4D+[26] with POS Pseudo Labels
Testing Set	UBFC-rPPG [22]
Metric (± Std. Err.)	MAE↓	RMSE↓	MAPE↓	ρ↑	SNR ↑
Supervised
TS-CAN [6]	4.69 ± 1.88	13.04 ± 100.15	4.51 ± 1.65	0.78 ± 0.10	0.01 ± 1.27
PhysNet(Normalized) [5]	1.78 ± 0.67	4.68 ± 11.94	1.92 ± 0.72	0.96 ± 0.04	1.24 ± 1.08
DeepPhys [35]	2.74 ± 0.96	6.78 ± 27.43	2.81 ± 0.91	0.93 ± 0.06	-0.22 ± 1.33
EfficientPhys-C [36]	2.43 ± 1.29	8.68 ± 67.51	2.52 ± 1.20	0.90 ± 0.07	0.39 ± 1.27
Training Set	BP4D+[26] with POS Pseudo Labels
Testing Set
Metric (± Std. Err.)	MAE↓	RMSE ↓	MAPE↓	ρ↑	SNR ↑
Supervised
TS-CAN [6]	1.29 ± 0.76	6.00 ± 33.74	1.60 ± 0.86	0.97 ± 0.03	8.61 ± 1.02
PhysNet(Normalized) [5]	3.69 ± 1.46	11.79 ± 64.42	7.35 ± 3.01	0.88 ± 0.06	8.33 ± 0.06
DeepPhys [35]	2.47 ± 1.41	11.11 ± 93.02	2.49 ± 1.21	0.89 ± 0.061	7.32 ± 1.09
EfficientPhys-C [36]	3.59 ± 1.84	14.55 ± 135.51	3.27 ± 1.50	0.80 ± 0.08	7.48 ± 1.15
MAE = Mean Absolute Error in HR estimation (Beats/Min), RMSE = Root Mean Square Error in HR estimation (Beats/Min), MAPE = Mean Percentage Error (\%),  $\rho =$  Pearson Correlation in HR estimation, SNR = Signal-to-Noise Ratio (dB) when comparing predicted spectrum to ground truth spectrum.
Table 11: Full 3-Fold Multitasking Results. For the BigSmall [41] method we show the full 3-fold results for multi-tasking PPG, respiration, and action unit classification; training and evaluating on the BP4D+ [26] (AU subset) dataset, using POS [31] derived pseudo training PPG labels.
Training Set
Testing Set
Fold	BP4D+[26]
BP4D+[26]
Fold 1	Fold 2	Fold 3
Heart
(Metric ± Std. Err.)	MAE↓	4.24 ± 0.73	2.91 ± 0.49	2.54 ± 0.48
RMSE↓	10.76 ± 33.20	7.26 ± 13.90	7.06 ± 16.67
MAPE↓	4.55 ± 0.74	3.22 ± 0.53	2.75 ± 0.49
ρ↑	0.68 ± 0.05	0.90 ± 0.03	0.91 ± 0.03
SNR↑	3.85 ± 0.69	6.27 ± 0.67	6.53 ± 0.63
Respiration
(Metric ± Std. Err.)	MAE↓	5.28 ± 0.31	4.96 ± 0.33	5.34 ± 0.35
RMSE↓	6.74 ± 4.38	6.67 ± 4.96	7.18 ± 5.12
MAPE↓	24.41 ± 1.55	25.30 ± 2.08	29.14 ± 2.72
ρ↑	0.15 ± 0.07	0.16 ± 0.72	0.12 ± 0.07
SNR↑	7.69 ± 0.64	10.53 ± 0.75	9.34 ± 0.64
Facial Action (AU)
(F1↑, Prec.)	AU01	18.62	11.04	18.88	11.34
AU02	20.76	12.73	18.28	10.89
AU04	12.57	8.08	11.48	7.85
AU06	66.73	66.58	64.71	61.09
AU07	74.86	78.68	70.08	75.10
AU10	74.92	77.32	70.09	74.48
AU12	72.69	70.79	67.75	68.54
AU14	67.21	72.84	70.18	69.11
AU15	22.56	13.91	22.33	13.38
AU17	25.77	18.01	20.95	12.45
AU23	34.64	27.41	34.21	24.19
AU24	7.00	3.71	10.70	6.20
Facial Action (AU)
(Metric Mean)	F1↑	41.53	39.97
Prec.↑	36.42	36.22
Acc. (%)↑	61.91	62.42
For HR estimation, MAE = Mean Absolute Error, RMSE = Root Mean Square Error, MAPE = Mean Percentage Error (%),  $\rho =$  Pearson Correlation, SNR = Signal-to-Noise Ratio (dB) when comparing predicted spectrum to ground truth spectrum. For AU classification F1 = harmonic mean of precision and recall, Prec. = precision, Acc. = accuracy.
H UBFC-Phys Video Exclusion
For evaluation of the UBFC-Phys [27] dataset in our main paper and by default in our toolbox, we utilized all three tasks and the same subject exclusion, or conversely sub-selection, list provided by the authors of the dataset in the second supplementary material of their paper [27]. Based on the aforementioned supplementary material, we eliminated 14 subjects (s3, s8, s9, s26, s28, s30, s31, s32, s33, s40, s52, s53, s54, s56) for the rest task (T1), 30 subjects (s1, s4, s6, s8, s9, s11, s12, s13, s14, s19, s21, s22, s25, s26, s27, s28, s31, s32, s33, s35, s38, s39, s41, s42, s45, s47, s48, s52, s53, s55) for the speech task (T2), and 23 subjects (s5, s8, s9, s10, s13, s14, s17, s22, s25, s26, s28, s30, s32, s33, s35, s37, s40, s47, s48, s49, s50, s52, s53) for the arithmetic task (T3).
In our toolbox, video exclusion is achieved using dataset filtering criteria specified in the config file. Specifically, an exclusion list or a task selection list can be provided to respectively exclude videos from being included or to select specific tasks as a part of a dataset.
I Multitasking Training and Evaluation Details
To show how this toolbox may be extended for physiological multitasking, we implement BigSmall [41] a model that multitasks PPG, respiration, and facial action. Here we reiterate information from [41], with slight modifications, for clarification.
I.1 Cross Validation Subject Folds
Fold 1: F003, F004, F005, F006, F009, F017, F022, F028, F029, F031, F032, F033, F038, F044, F047, F048, F052, F053, F055, F051, F061, F063, F067, F068, F074, F075, F076, F081, M003, M005, M006, M009, M012, M019, M025, M026, M028, M031, M036, M037, M040, M046, M047, M049, M051, M054, M056.
Fold 2: F001, F002, F008, F018, F021, F025, F026, F035, F036, F037, F039, F040, F041, F042, F046, F049, F057, F058, F060, F062, F064, F066, F070, F071, F072, F073, F077, F082, M001, M002, M007, M013, M014, M016, M022, M023, M024, M027, M029, M030, M034, M035, M041, M042, M043, M048, M055.
Fold 3: F007, F010, F011, F012, F013, F014, F015, F016, F019, F020, F023, F024, F027, F030, F034, F043, F045, F050, F051, F054, F056, F059, F065, F069, F078, F079, F080, M004, M008, M010, M011, M015, M017, M018, M020, M021, M032, M033, M038, M039, M044, M045, M050, M052, M053, M057, M058.
I.2 AU Subset
The AU subset used for model training and evaluation (in this toolbox) is made up of dataset subset which contains action unit labels. This consists of approximately 20 seconds worth of data from the following tasks for each subject: T1, T6, T7, T8.
I.3 Subject Fold Splits
[41] is evaluated using 3 fold cross validation, where the folds are comprised of trials from mutually exclusive subjects in the dataset. These subject-wise folds are outlined below.
J Additional Features
J.1 Pre-processed Data Visualization
Pre-processing is an important aspect of the rPPG task that we hope to help standardize using our toolbox. It is advantageous to be able to quickly visualize and visually evaluate pre-processed image data and ground truth signals. Image data in particular can be especially useful to observe in order to inspect the effectiveness of out-of-the-box face detection and cropping techniques used in our toolbox, and to ultimately get an idea as to how much of the face region is visible in a given video. We provide simple Jupyter Notebooks for quickly visualizing image data and ground truth signals pre-processed by our toolbox. Further details regarding these notebooks can be found in our GitHub repo and the associated README.
J.2 Training Loss, Validation Loss, and Learning Rate Visualization
The rPPG-Toolbox assumes certain defaults across most config files for supervised methods, including a default learning rate of 0.009 used alongside the Adam [38] or AdamW [39] optimizers, a criterion such as mean squared error (MSE) loss or Negative Pearson Correlation Loss, and the 1 cycle learning rate scheduler [40] are utilized for training. An exception is with BigSmall [41], which uses a default learning rate of 0.001 that remains constant throughout training. It can be valuable to visualize losses such as those involved in training or validation phases. Furthermore, it may be useful to simultaneously visualize the learning rate, especially when users stray from the defaults in order to target an optimal set of training, validation, and testing parameters for their research efforts. The toolbox's config contains parameters that enable the visualization of the training loss, validation loss, and the learning rate for any given supervised method.
J.3 Bland-Altman Plots
We provide Bland-Altman plots as an additional metric in the rPPG-Toolbox. Users can enable the plots via an evaluation parameter in the config file, and will be given further options to configure the plots as the toolbox is refined and expanded. For more details, please refer to the GitHub repo and the associated README.
J.4 Motion Analysis
We also provide scripts that leverage OpenFace [44] for extracting, visualizing, and analyzing motion in rPPG video datasets. Specifically, we include a Python script to convert datasets into the .mp4 format for subsequent analysis by OpenFace, a shell script that leverages OpenFace to perform both rigid and non-rigid head motion analysis, and a separate Python script that plots exemplar plots that showcase comparisons of motion between different datasets. Further details can be found in our GitHub repo and the associated README.