Learning Higher-Order Dynamics in Video-Based Cardiac Measurement
Authors: Brian L. Hill
Xin Liu
Daniel McDuff
Venue: ICLR 2022 Submitted (2022)
Keywords: Computer Vision
Dynamic Systems
Deep Learning

Abstract: Computer vision methods typically optimize for first-order dynamics (e.g., optical flow). However, in many cases the properties of interest are subtle variations in higher-order changes, such as acceleration. This is true in the cardiac pulse, where the second derivative can be used as an indicator of blood pressure and arterial disease. Recent developments in camera-based vital sign measurement have shown that cardiac measurements can be recovered with impressive accuracy from videos; however, the majority of research has focused on extracting summary statistics such as heart rate. Less emphasis has been put on the accuracy of waveform morphology that is necessary for many clinically impactful scenarios. In this work, we provide evidence that higher-order dynamics are better estimated by neural models when explicitly optimized for in the loss function. Furthermore, adding second-derivative inputs also improves performance when estimating second-order dynamics. By incorporating the second derivative of both the input frames and the target vital sign signals into the training procedure, our model is better able to estimate left ventricle ejection time intervals.

Under review as a conference paper at ICLR 2022
LEARNING HIGHER-ORDER DYNAMICS IN VIDEO- BASED CARDIAC MEASUREMENT
Anonymous authors Paper under double-blind review
ABSTRACT
Computer vision methods typically optimize for first-order dynamics (e.g., opti- cal flow). However, in many cases the properties of interest are subtle variations in higher-order changes, such acceleration. This is true in the cardiac pulse, where the second derivative can be used as an indicator of blood pressure and arterial disease. Recent developments in camera-based vital sign measurement have shown that cardiac measurements can be recovered with impressive accu- racy from videos; however, the majority of research has focused on extracting summary statistics such as heart rate. Less emphasis has been put on the ac- curacy of waveform morphology that is necessary for many clinically impactful scenarios. In this work, we provide evidence that higher-order dynamics are bet- ter estimated by neural models when explicitly optimized for the loss function. Furthermore, adding -econd-derivative inputs also improves performance when estimating accond-order dynamics. By incorporating the second derivative of both the input frames and the target vital sign signals into the training procedure, model is better able to estimate left ventricle ejection time (LVET) intervals.
INTRODUCTION
Many of the properties of dynamical systems only become apparent when they move or change as the result of forces applied them. In most applications we are interested in behavior in terms of positions, velocities, and accelerations, and in some cases the properties of interest may only be ob- served in subtle variations in the higher-order dynamics (e.g., acceleration). Whether monitoring the flight of a drone to create control mechanism for stabilization or analyzing the fluid dynamics of the cardiovascular system in the human body, there can be a need recover these dynamics accurately. However, most video-based systems are trained on lower-order signals, such as position in the case of landmark tracking or velocity/rate-of-chapg
(optical flow) the case of visual odometry (Nister et al. 2004). Thus, they optimize for lower (zeroth or first) order dynamics. Does this harm their ability to estimate higher order changes? We hypothesize that networks trained to predict temporal signals will benefit from combined multi-derivative learning objectives. To test this hypothesis, explore rideo-based cardiac measurement as an example application with a complex dynamical sys- tem (the cardiovascular system) and introduce simple but effective changes the inputs and outputs to significantly improve the measurement of clinically relevant parameters.
Photoplethysmoograhyy (PPG) is a low-cost and non-invasive method for measuring the cardiovas- cular blood volume pulse (BVP). There are many clinical applications for PPG as the signal contains substantial information about health state and risk of cardiovascular diseases (Elgendi et al. 2019 Reisner et al. 2008 Pereira et al. 2020). In the current world, an acutely relevant application of PPG is for pulse oximetry (i.e. measuring pulse rate and blood oxygen saturation) as can be used to detect low blood oxygen levels associated with the onset of COVID-19 (Grrennaalgh et al.) 2021 The COVID-19 pandemic has accelerated the adoption of telcheath systems (Annis et al. 2020) with more and more clinical consultations being conducted virtually. Therefore, techniques for remotely monitoring physiological vital signs are becoming increasingly important (Gawatko et al., 2021 Rohmetra al. 2021). As one might expect, with many clinical applications the precision with which the PPG signal can be recovered is critical importance when it comes to accurate inference of downstream conditions and the confidence of practitioners in the technology.

---

Under review as a conference paper at ICLR 2022
To date, in video- PPG measurement the primary focus of analysis and evaluation has been on features extracted from the raw waveform or its first derivative (Chen & McDuff 2018 Liu et al. 2020 2021; Poh et al. 2010a). However, the second derivative of the PPG signal highlights subtle features that can be difficult to discern from those in the lower derivatives. Since the second deriva- tive reflects the acceleration (Takazawa] 1993) the rate-of rate-of change of the blood volume, it is more closely related to the change in pressure applied by the heart on blood vessels and its relation to vascular health.
An example of a particular feature accentuated in the second-derivative (i.e. acceleration) PPG is the dicrotic notch (see Fig. D. which occurs when the heart's aortic valve closes due to the pressure gradient between the aorta and the left ventricle. The dicrotic notch may only manifest as an in- flection in the raw PPG wave; however, in the second derivative this inflection a maxima. [Inoue et al. (017) found that the second derivative of the PPG signal can be used as an indicator of arte- rial stiffness which itself an indicator of cardiac disease. [akazawa et al. (1998) evaluated the second derivative of the PPG waveform and found that its characteristic shape can be used to esti- mate vascular aging, which was higher in subjects with a history of diabetes mellitus, hypertension, hypercholesterolemiaa, and ischemic heart disease compared
age-matched subjects without.
While the second derivative of a signal can be a rich source of information, often the zeroth- or first-order dynamies are given priority. For example, Chen & McDuff (2018) observed that training video- or imaging-based PPG (iPPG) models using first-derivtiie (difference) frames as input with an objective funetion of minimizing the mean squared error between the prediction and the first derivative of the target BVP signal was effective. This approach was used because the authors were designing their system measure systolic time intervals only, which are most prominent in the lower order signals. However, they did not combine this with higher-order derivatives nor did they do any systematic comparison across derivative objectives.
We argue that model trained with an explicit duce feature representations that better preserve/recover these dynamics than methods that simply derive acceleration from velocity. We observe that providing the model with a second derivative input also helps the network to better predict both the first and second derivative signals.
-erivvivvvvvivee (acceleration) objective should pro-
Finally, as diverse labeled data for training supervised models for predicting dynamical signals is often difficult to come by, we build on promising work in simulation to obtain our training data. Since light is absorbed and reflected differently for different skin tones (Bent et al. 2020) Dasari jct al. (22TT) having a training set that represents the true diversity of the target population is crucial for sufficient generalization. Our results show that models trained with synthetic data can learn parameters that successfully generalize to real human subjects. While this is not a central focus of our paper, we believe that it presents a promising proof-of-concept for future work.
To summarize, in this paper, we 1) demonstrate that directly incorporating higher-order dynamics into the loss function improves the quality of the estimated higher-order signals in terms of waveform morphology, 2) show that adding second-derivative inputs additionally improves performance, and 3) we describe novel deep learning architecture that incorporates the second derivative input frames and target signals and evaluate it against clinical-gras contact sensor measurements.
2 BACKGROUND
Learning Higher-Order Motion from Videos. Despite its significance in many tasks, acceleration is often not explicitly modeled in many computer vision methods. However, there is small body of literature that has considered how to recover (Edison Jijil 2017) and amplify optical accelera- tion (Zhagg et al. 2017 Takeda et al.) 2018). Given that acceleration can be equally as important as position and velocity understanding dynamical systems, we argue that this topic deserves further attention.
A particularly relevant problem to ours is identifying small changes in videos (Wu et al. 2012 Zhang et al.) 2017 Chen & McDuff 2020; Takeda et al., 2018), and specifically in acceleration in the presence of relatively large motion. As an example, in the iPPG prediction task the aim is to identify minor changes in skin coloring due to variation in blood flow patters,, while ignoring major pixel changes due to subject or background motion. One method proposed by Zhang et al. 2017) for overcoming this signal separation problem is Video Acceleration Magnification, in which
2

---

Under review as a conference paper at ICLR 2022
large motions are assumed to be linear the temporal scale of small changes while small changes deviate from this linearity. An extension to this method focused on making it more robust to sudden motions (Takeda et al., 2018). In both cases, combination of Eulerian and Lagrangian approaches was used, rather than utilizing a supervised learning paradigm. Of relevance here is also work magnifying subtle physiological changes using neural architectures (Chen & McDufi (020). which have been shown to effectively separate signal and noise in both the spatial and temporal domains.
Our work might be most closely related prior research into feature descriptors for optical accel- eration (Edison & Jijil 2017). One example uses histograms of optical acceleration to effectively encode the motion information. However, this work also defined handcrafted features, rather than learning representations from data. Our work is also related conceptually to architectures such as Flst (Feichtenhofer et al., 2019) in that it utilizes multiple "pathways" to learn different proper- ties of the dynamics within a video. We were inspired by this approach; however, unlike SlowFast, we focus specifically on higher-order pathways rather than slower and faster frame sequences.
Video-based Cardiac Measurement. Diffuse reflections from the body vary depending on how much light is absorbed in the peripheral layers the skin and this influenced by the volume of blood in the capillaries. Digital cameras can capture these very subtle changes in light which can then be used to recover the PPG signal (Wu et al. 2000 Takano & Ohtal 2007, [Verkrussse] ct al. 2008 Poh et al. 2010a). The task then becomes separating pixel changes due blood flow from those due to body motions, ambient lighting variation, and other environmental factors that we consider noise in this context. While earlier methods leveraged source separation algorithms (Wang) iet al. 2016), such as ICA (Poh et al. 2010a) or PCA (Lewandowska et al. 2011). neural models provide the current state-of-the-arr in this domain (Chen & McDufi, 2018 Liu al. 2020 [202]
Song et al. [02TT] Lu et al. 2021). These architectures support learning spatial attention and source-
specific temporal variations and separating these from various sources of noise. Typically, the input to these models are normalized video frames and the output is a I-D time series prediction of the PPG waveform or the heart rate. vast majority of work has evaluated these methods based erross in heart rate estimation, which considers the dominant or "systolic" frequency alone. Only few papers have used more challenging evaluation criteria, such as the estimation systolic to diastolic peaks (McDuff et al. 2014).
OPTICAL BASIS
We start by providing an optical basis for the measurement of the pulse wave using a camera and specifically the second derivative signal. Starting with Shafer's Dichromatic Reflection Model (DRM) (Wang et al., 2016 Chen & McDuff, 2018 Liu et al., 2020), we want to understand how higher order changes in the blood volume pulse impact pixel intensities to motivate the design of our inputs and loss function. Based on the DRM model the RGB values captured by the cameras as given by:
(t) = I(t) (v. + va(()) + (t)
where (t) is the luminance intensity level, modulated by the specular reflection v,(t) and the dif- fuse reflection va(t). Quantization noise of the camera sensor is captured by v, (t). T(t) can be decomposed into stationary and tire-ng parts v. (t) and va(t) (Wang et al. 2016):
Va = un do + up p(t)
(2)
where u2 is the unit color vector of the skin-tissue; dn is the stationary reflection strength; up is the relative pulsatile strengths caused by hemoglobin and melanin absorption; p(() represents the phys- iological changes. Let us assume for simplicity in this case that the luminance, / (i.e., illumination in the video) is constant, not time varying, which is a reasonable assumption for short vidcos and those in which the subject can control their environment (c.g., indoors). Then differentiating twice with respect to time, t:
8'Cr(1)
= Pvu((!) 012
Seus ōt
(u,(1))
Por
+
(3)
0t
01
012
The non-time varying part Ua do becomes zero. Thus simplifying the equation to:
3

---

Under review as a conference paper at ICLR 2022
(t) 0t
uup(t)
(t)(())
=I.
Ot2
ōt2
Ot2
Furthermore, if specular reflections do not vary time (e.g.. if the camera and subject are station- ary), the (I) term will also become zero. This means that the second derivative changes in pixel intensities
of second derivative changes in PPG and camera noise. With current camera technology, and little video compression, image noise is typically much smaller than the PPG signal. Therefore, we would expect the pixel changes to dominated by second derivative variations the blood volume pulse:
Ct) 0t2
1 (t)((t) 0t2
(5)
As such, we can infer that when attempting to estimate the second derivative of the PPG signal from videos without very large motions illumination changes, second derivative changes in the pixel space would appear helpful and that minimizing the loss between the second derivative prediction and ground truth will be the simplest learning task for the algorithm when the input is second- derivative pixel changes.
OUR MODEL.
To test our hypothesis we incorporate higher- order dynamics into model via the loss func- tion and as inputs to see if they will pro- vide better estimation of the higher-order dynamics of the target signal. We pro- pose a multi-derivative convolutional attention network architecture (see Fig. 2 that oper- ates separately on each set of derivative in- put frames to extract sequences of features, and then maps these sequences to the target derivative signals. The convolutional atten- tion network (CAN) paradigm was first de- scribed by Chen & McDuff (2018) to cap- ture the tional operations and uses soft-attention masks to help segment the subject from the back- ground and focus on relevant regions of body. In our case, we used a CAN architec- ture with three branches: (1) a first-derivative branch, (2) second-derivative branch, and (3) an attention branch. The first-derivative branch extracts features from the differences between consecutive video frames, and similarly the second-derivative branch that extracts features from the differeecceoofdiffeeee
Left Ventricle Ejection Time (LVET)
Systolic Peak
BVP
Dicratic Norch
Derivative
Systolic Foot
Second Derivative
changes with convolu-
time (9)
Figure I: The Left Ventricle Ejection Time (LVET) is the duration between the beginning and end of the systolic phase. This interval corre- sponds to the opening and closing of the heart's aortic valve, during which the left ventricle ejects blood into the system. the PPG waveform, this interval begins at the diastolic point and ends with the dicrotic notch.
frames. The attention branch uses the raw video frames to learn attention masks (one per frame) that en- courage the network to prioritize regions of the image that contain useful signal (e.g. partici- pant's skin) and ignore noisy regions (e.g. background). These attention masks are shared between the first-derivative branch and the second-derivative branch to contain first and second derivative information. After feature representations are extracted from frames within each derivative-ing input branch, the features concatenated together for each time step and the target signals are then generated using recurrent neural network (RNN) layers. A diagram depicting the architecture used for experimentation is shown in Fig.
expect the same spatial regions
4

---

Under review as a conference paper at ICLR 2022
Second Derts. loput
30 Cana
30 38xd5
30 Corr
au Comn, 3D Gom..
Second Derly, Loss (MAR]
38c05
15xtd
T8x15
TxT
RNW
Finst Darts Input
PINN
Pirsi Deriy, Loss (MAE)
from First (Base
Secord Calculare
[w1
Ti
TwS
lopur
Mic3
30:30
30+36
10x1
18x10
18c1
Appesminn
Figure 2: Our multi-derivative architecture used for experimentation. Spatial features extracted separately for each set of derivative frames using 3D convolutional layers and mean pooling lay- ers. Once feature representations catenated together and recurrent layers are used for modeling the temporal signals. The first- and second-derivative losses are calculated as the mean absolute error between the predictions and the synchronized ground-truth PPG signal.
extracted, the temporal features from each branch are con-
4.1 PREDICTING MULTI-DERIVATIVE TARGET SIGNALS
The goal of iPPG is to obtain an estimate of the underlying PPG signal p(t) (as in Eq 2), while only observing video frames X (t) containing a subject's skin (in this case the face). Mathematically, this can be described as learning function: p(t) = (t) because we are interested in changes in blood volume changes, estimating the first derivative of the PPG signal: (t) = (t)
where the first derivative PPG signal is defined as: (t) p(t) ppt 1). Using prior methods, to obtain an estimate of the PPG signal's s second derivative, one would either differentiate the predicted PPG signal twice, differentiate the predicted destiverivv PPG once, rather than caleulate the acceleration PPG directly. In contrast, we explicitly predict the acceleration PPG waveform as a target signal. We define the second derivative waveform the difference between consecutive first-derivative time points: y²(1) p' (4) - p'(1 - 1). Then we train our model to predict the second derivative waveform 5t (t),)), (t) given set of input video frames X (t) and the corresponding normalized difference frames X')).. To optimize our model parameters we minimize the mean squared difference between the true and predicted second derivative waveforms:
Σ (V((1) -"
(6)
4.2 LEVERAGING MULTI-DERIVATIN INPUTS
has been previously shown that the normalized difference frames are useful for predicting the first derivative PPG waveforms. Therefore, we hypothesized that incorporating the second derivative of the raw video frames X X"()) X'(t) Xt 1) (i.e. the difference-oo-difffreeee frames) may also be useful for predicting the PPG signal and its derivatives. Similar to the difference frames, we added a separate convolutional attention branch, where the attention mask shared between both branches (see Fig. 2b. Sharing the attention mask is reasonable assumption as we would expect skin regions to exhibit the signal and similar dynamies. After the feature maps in each branch are pooled into single value per feature at each time step, the learned representations together. These concatenated features over time are used as input sequences to the recurrent layers that generate the target waveforms.
concatenated
Given that difference frames (t) are useful for predicting the first derivative PPG waveforms, features learned from the cefrence-oo-diffeeeee frames X"(() may be beneficial for predicting the second derivative PPG signal. theory, if siaecrenccoo--dfffeeeee features are indeed useful for predicting the acceleration PPG, then the CAN network should be able to learn those features
5

---

Under review as a conference paper at ICLR 2022
True PPG
FD-optimized Prediction
SD- SD-optimized Prediction
Normalized Raw PPG Amplitude
1
2
4
4-
Normalized 1st Derivative PPG Amplitude
2
-2
2
4
Normalized 2nd Derivative
3-
2
PPG Amplitude
-1
-2
-3
Time (seconds)
Figure 3: Comparison of true (black) and predicted (blue and green) raw zeroth order (top), first order derivative (middle), and second order derivative (bottom) waveforms. The blue and green lines reflect two models: predicting the first derivative (blue) and predicting the second derivative (green). Diastolic points are labeled with red dots, and dicrotic notches are labeled with blue dots. LVET intervals are labeled by dotted red lines. Notice how the points of interest are generally more obvious in the second derivative waveforms, as they are maxima rather than inflections. Also note that the LVET time intervals for the second derivative model are generally similar those from the contact (true) PPG.
from the difference frames due to the 3D convolutional operations. However, manually adding the Sefeereencc-ofddffeeeeee frames could help guide the model. To examine the effect of combining higher-order inputs and target signals, we fit a model p²(t)) = f(X(t), (()) to predict the second-derivative PPG.
EXPERIMENTS
In this section we will describe the data used to train and evaluate our method and perform sys- tematic ablation study in which test different combinations of inputs and outputs.

---

Under review as a conference paper at ICLR 2022
300
FD-optimized Predictfons
20
FD-optimized Predictions
SD SD-optimized Predictions
SD SD-optimized Predictions
250
Absolute Error (BPM)
Absolute LVET Error (ms)
15
200
10
150
100
50
40
70 True HR (BPM)
85
100
125
175
225 True LVET (ms)
275
325
Figure 4: Bland-Altman plots comparing error distributions for average task heart rate (left) and LVET intervals (right) for the model optimized for firt-derivative prediction (blue) and the model optimized for -econd-derivative prediction (green). (left) The absolute difference between the true and predicted heart rate for each subject/task. (right) The absolute difference between the predicted and true values (y-axis, in milliseconds) is plotted as function of the true LVET (x-axis, in millisec- onds). The solid line represents the mean error, and the dotted lines represent the 95% confidence intervals (+ 1.96 standard deviation).
5.1 DATA
Training To train our models using large and diverse set of subjects, we leverage recent work that uses highly-parameterizae synthetic avatars to generate videos containing simulated subjects with various movements and backgrounds (McDuff et al. 2020). To drive changes in the synthetic avatars' appearance, the PPG signal is used manipulate the base skin color and the subsurface radius (McDuff al. 2020). The subsurface scattering is spatially weighted using an artist-created subsurface scattering radius texture that captures variations in the thickness of the skin across face. Using physiological waveforms signals from the MIMIC Physionet (Goldberger Ary L. al. 2000) database, we randomly sampled windows PPG waveforms from real patients. The phys- iological waveform data synthetic avatar pipeline and MIMIC waveforms, we generated 2,800 6-second videos, where half of the videos were generated using hand-crafted facial motion/action signals, and the other half ing facial motion/action signals extracted using landmark detection on real videos. Examples of the avatars be found in Appendix A.I.I
sampled to maximize examples from different patients. Using the
Testing Given that we are focusing on recovering very subtle changes in pixel intensities due to the blood volume pulse, we use highly controlled and very accurately annotated dataset of real videos for evaluation. The AFRL dataset (Estepp et al., (014)) consists of 300 videos from 25 participants (17 male and 8 female). Each video the dataset Hz. Ground truth PPG signals were recorded using contact reflective PPG sensor attached to the subject's index finger. Each participant was instructed to perform three head motion tasks including rotating the head along the horizontal axis, rotating the head along the vertical axis, and rotating the head randomly once every second to one of nine predefined locations. Since our goal in this work was compare methods for estimating subtle waveform dynamics, which can be more difficult to do in the presence large motion, we focused here on the first two AFRL tasks where participant motion minimal. Examples of AFRL participants be found in Appendix A.I.I
resolution of 658x492 pixels sampled at 30

---

Under review as a conference paper at ICLR 2022
Table 1: Quantitative cross-dataset performance comparison between different architecture config- urations on the AFRL dataset. Models were trained using only the synthetic dataset. Values shown are (mean + standard deviation).
beats-per-minute (BPM); First Derivative (FD); Heart Rate (HR); Mean Absolute Error (MAE);
Second Derivative (SD); Left Ventricle Ejection Time (LVET).
Input Frames
Target Signals
HR MAE (BPM)
LVET MAE (ms)
FD
SD
FD
SD
(FD- (FD-Optimized)
X
0.66 + 2.07
108.26 + 56.19
X
/
V
1.63 + 4.21
64.77 = 31.04
/
X
V
1.68 = 4.85
57.29 22.68
V
J
J
X
0.88 2.75
75.41 + 43.47
J
V
1.16 + 3.67
65.99 31.88
(SD-Optimized)
X
V
3.41 + 8.77
52.07 + 19.53
5.2 IMPLEMENTATION DETAILS
We trained our models using a large dataset of generated synthetic avatars and evaluated model performance on the AFRL dataset, which consists of real human subjects. For each video, we first cropped the video frames so that the face was approximately centered. Next, we reduced the resolu- tion of the video 36x36 pixels to reduce noise and computational requirements while maintaining useful spatial signal Verkruysse et al. (2008); Wang et al. (2017); : Poh et al. (2010b). The input to the attention branch was 7 raw video frames. The input to the first-derivative branch was set of T normalized difference frames, calculated by subtracting consecutive frames and normalizing by the sum. The imput to the second-derivative branch was set of T difference-of fafeerencoooo-dfffeeenee frames (second derivative frames), calculated by subtracting consecutive normalized difference frames (i.e. the T frames used input the motion branch). In our experiments, of T = 30 video frames to predict the target signals for the corresponding 30 time points, Dur- ing training, a sliding window of 15 frames (i.e. 50% overlap between consecutive windows) was used to increase the total number of training examples. The model was implemented using Ten- sorflow (Abadi al.) 2016) and trained for eight epochs using the Adam (Kingma & Ba 2017) optimizer with a learming rate of 0.001, and a batch size of 16.
used a window size
5.3 SYSTEMATIC EVALUATION
To measure the effect of using multi-derivative inputs and outputs, we systematically removed the second-derivative parts of the model and used quantitative and qualitative methods to examine the change in model performance. To quantitatively measure the quality of the predicted signal, calculated two clinically important parameters heart rate (HR) ) and the left ventricular ejection time (LVET) interval (see Appendix 1.113 for details). Video-based HR prediction has been major focus of iPPG applications, with many methods showing highly-accurate results. HR can be determined through peak detection or by determining the dominant frequency in the signal (c.g. using fast Fourier transform). Since current iPPG methods are able to achieve sufficiently-low error rates on the HR estimation task, we believe that metrics that capture the quality of waveform morphology should also be considered.
The LVET interval is defined the time between the opening and closing of the heart's aortic valve, iee the systolic phase when the heart is contracting (see Fig. In the PPG waveform, this interval begins at the diastolic point (i.e. the global minimum pressure within heartbeat cycle) and ends with the dicrotic notch (i.c. local minimum occurring after systolic peak, marking the end of the systolic phase and the beginning of the diastolic phase). LVET typically is correlated with cardiac output (stroke volume x heart (Hamada et al. 1990). and has been shown to be an indicator of future heart failure as the time interval decreases with left-ventricle dysfunction Shierig-SSrennse et al. 2018)
Calculating LVET requires identification of the diastolic point and the dicrotic notch. The diastolic point is (global) minimum point within heart beat, meaning corresponds to positive peak
8

---

Under review as a conference paper at ICLR 2022
in the second derivative signal according to the second-derivative test. Similarly, the dicrotic notch is a (local) minimum in the PPG signal, and appears as a positive peak in the second derivative following the diastolic peak in time. Because the dicrotic notch can often be subtle feature, it is much easier to identify in the PPG's second derivative compared to the raw signal. Therefore, is good example of dinically-importan waveform morphology that is best captured by higher-order dynamics.
Removing the second-derivatee frames In Table | quantitative evaluation metrics (HR and LVET) are shown for all experiments in our ablation study, using tasks I and 2 from the AFRL dataset. Removing the second-derivative (SD) frames results in the model configurations in the top three rows of Table [I] When SD frames are removed, the result is a general decrease in the HR error. However, there is also a general increase in LVET interval prediction error, which suggests that including the SD frames leads to improved estimation of waveform morphology.
Removing the first-dervatiie target signal Intuitively, models that are optimized using loss function specifically focusing on single objective will perform better in terms of that objective compared to models trained with loss functions containing multiple objectives. By removing the first-derivative target signal from the training objective, the model is focused to exclusively focus on the second-derivatio the lowest LVET MAE of any model configuration (last row of T. While the SD-Optimized model achieves the lowest LVET error, the HR error is the highest of any configuration. These results suggest that there are performance trade-offs to consider when designing system for particular downstream tasks.
(SD) objective. Empirically, this leads the SD-Optimized model to have
Removing the aecond-derivative target signal When the second-derivative target signal is removed from the model, the optimization procedure is purely focused on improving the prediction of the first derivative. The FD-Optimized model (first row of Table serves as a form of baseline, since previous works have focused on using first-derivtiive (FD) frames to predict the first-derivative PPG signal. Fig. shoowss a Bland-Altman plot (Martin Bland & Altman 1986) comparing the FD- Optimized and SD--Otiimzed error distributions as a function of the ground-truth values both HR and LVET intervals.
Perhaps unsurprisingly, our results show the FD-Optimized model achieves the lowest HR MAE (0.66 + 2.07 BPM) of any model configuration examined and, in particular, improves HR esti- mation compared to models without the first derivative target signal. However, the FD-Optimized model also has the worst performance in terms of the LVET MAE (108.26 + 56. ms) of any model configuration. This suggests that while the configuration provides accurate assessment of the heartbeat frequency, the quality of predicted waveform morphology can be improved by incor- porating dervndeder information. We observe similar results when evaluating the models on the UBFC (Bobbia et al. 2019) and PURE (Stricker et al. 2014) datasets (see Appendix Table 3)
Qualitative comparisons For a qualitative comparison, in Fig. Optimized, and SD-Optimized PPG, first derivative, and second derivative. Additionally, in the bot- tom panel of Fig. 3 we overlay the true and predicted LVET intervals for each signal to demonstrate model performance. For additional qualitative comparisons, see Appendix A.2
plot the ground-truth FD-
CONCLUSIONS
Using the task of video-based cardiac measurement we have shown that when learning representa- tions for dynamical systems that appropriately designing inputs, and optimizing for derivatives of interest can make a significant difference in model performance. Specifically, there is trade-off between optimizing for lower-order and higher-order dynamics. Given the importance of second- derivatives (i.e., acceleration) in this, and many other video understanding tasks, we believe it is important to understand the trade-off between optimizing for targets that capture different dynamic properties. In cardiac measurement in particular, the LVET is one of the more important clinical parameters and can be better estimated using higher-order information. While we have investigated the importance of higher-order dynamics in the context of tideo-based cardiac measurement, this paradigm is generally applicable. We believe future work will continue to showcase the importance of explicitly incorporating higher-order dynamics.

---

Under review as a conference paper at ICLR 2022
7 ETHICS STATEMENT
Camera-based cardiac measurement could help improve the quality of remote health care, as well as enable less invasive measurement of important physiological signals. The COVID-19 pandemic has revealed the importance of tools to support remote care. These needs are likely to be particularly acute low-resource settings where distance, travel costs, and time are great barrier to access quality healthcare. However, given the non non-contact nature of the technology. it could also be used to measure personal data without the knowledge of the subject. Just as is the case with traditional con- tact sensors, it must be made transparent when these methods are being used, and subjects should be required to consent before physiological data is measured or recorded. There should be no penalty for individuals who decline to be measured. New bio-metrics laws can help protect people from un- wanted physiological monitoring, or discrimination based on pre-existing health conditions detected via non-contact monitoring. However, social norms also need be constructed around the of this technology.
In this work, data were collected under informed consent from the participants.
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving. Michael Isard, Yangging Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vin- cent Vanhoucke, Vijay Vasudevan, Fernanda Viegas, Oriol Vinyals, Pete Warden, Martin Watten- berg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. arXiv:1603.cmm 0400404447740 [cs], March 2016. arXiv: 1603.04467.
Tucker Annis, Susan Pleasants, Gretchen Hultman, Elizabeth Lindemann, Joshua A Thompson, Stephanie Billecke, Sameer Badlani, and Genevieve B Melton. Rapid implementation of COVID-19 remote patient monitoring program. Journal of the American Medical Informatics Association, 2768):122--1300, August 2020.
Brinnae Bent, Benjamin A. Goldstein, Warren Kibbe, and Jessilyn Dunn, Investigating sources of inaccuracy in wearable optical heart rate sensors. upj Digital Medicine, 3(1):1-9, February 2020. Number: Publisher: Nature Publishing Group.
Tor Biering-Sorensen, Gabriela Querejeta Roca, Sheila M. Hegde, Amil M. Shah, Brian Claggett, Thomas H. Mosley, Kenneth R. Butler, and Scott D. Solomon. Left Ventricular Ejection Time is an Independent Predictor of Incident Heart Failure in a Community based Cohort. European (007):1106-110 journal of heart failure, 20(7): 106- 1114, July 2018.
Serge Bobbia, Richard Macwan, Yannick Benezeth, Alamin Mansouri, and Julien Dubois. Unsu- pervised skin tissue segmentation for remote photoplethysmographyy Pattern Recognition Letters, 124:82-90, June 2019.
Weixuan Chen and Daniel McDuff. Deepphys: Video-based physiological measurement using con- volutional attention networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 349-365, 2018.
Weixuan Chen and Daniel McDuff. DeepMag: Source-Specific Change Magnification Using Gra- dient Ascent. ACM Transactions Graphics, 40(1): September 2020.
Ananyananda Dasari, Sakthi Kumar Arul Prakash, László A. Jeni, and Conrad S. Tucker. Eval- uation of biases in remote photoplethysmography methods. npj Digital Medicine, 4(1):1-13, Cc_liceese_tyre _tppe: cc_by Cg.type: Nature Research Journals Num- ber: Srimary-atyper Research Publisher: Nature Publishing Group Subject.term: Biomark- erv:eevelopnng world Subject_term id: biomarkersddevlooingg-
June 2021. Bandiera_abtest
Anitha Edison and CV Jiji. Optical acceleration for motion description in videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 39-47, 2017.
Mohamed Elgendi, Richard Fletcher, Yongbo Liang, Newton Howard, Nigel H. Lovell, Derek Ab- bott, Kenneth Lim, and Rabab Ward. The use of photoplethysmogrrphy for assessing hyperten- sion. npj Digital Medicine, 2(1))60, June 2019.
Justin R. Estepp, Ethan B. Blackford, and Christopher M. Meier. Recovering pulse rate during motion artifact with a multi-imager array for non-contact imaging photoplethysmograhhy, In 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 1462-1469, October 2014. ISSN: 1062-922X.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6202-6211, 2019.
Monika Gawalko, David Duncker, Martin Manninger, Rachel M.J. van der Velden, Astrid N.L. Hermans, Dominique V.M. Verhaert, Laurent Pison, Ron Pisters, Martin Hemels, Arian Sultan, Daniel Steven, Dhiraj Gupta, Hein Heidbuchel, Afzal Sohaib, Petra Wijtvliet, Robert Tieleman, Henri Gruwez, Julian Chun, Boris Schmidt, John J. Keaney, Patrick Müller, Piotr Lodziński, Emma Svennberg, Olga Hoekstra, Ward P.J. Jansen, Lien Desteghe, Tom de Potter, David R.
11

---

Under review as a conference paper at ICLR 2022
Tomlinson, Lis Neubeck, Harry J.G.M. Crijns, Nikki A.H.A. Pluymackers, Jeroen M. Hendriks, Dominik Linz, and the TeleCheck-AF investigators. The European TeleCheck-AF project on remote app-based management of atrial fibrillation during the COVID-19 pandemic: centre and patient experiences. EP Europace, 23(7):1003-1015, July 2021.
Goldberger Ary L., Amaral Luis A. N., Glass Leon, Hausdorff Jeffrey M., Ivanov Plamen Ch., Mark Roger G., Mietus Joseph E., Moody George B., Peng Chung-Kang and Stanley H. Eugene. PhysioBank, PhysioToolkin, and PhysioNet. Circulation, 101(23):211-=2220 June 2000.
Trisha Greenhalgh, Matthew Knight, Matt mima--Kim, Naomi J. Fulop. Jonathan Leach, and Cecilia Vindrola-Padros Remote management of covid-19 using home pulse oximetry and virtual ward Publisher: British Medical Journal Publishing Group
support. BMJ, 372:n677, March 2021.
Section: Practice.
M. Hamada, K. Hiwada, and T. Kokubu, Clinical significance of systolic time intervals in hyperten- sive patients. European Heart Journal, 11 Suppl 1:105-113, -13,, December 1990.
Noriko Inoue, Hideshi Kawakami, Hideya Yamamoto, Chikako Ito, Saeko Fujiwara, Hideo Sasaki, and Yasuki Kihara. Second derivative of the finger photoplethysmogran and cardiovascular mortality middle-aged and elderly Japanese women. Hypertension Research, 40(2):207-211, February 2017.
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. 14Xiv::412.690 [cs], January 2017, arXiv: 1412.6980.
Magdalena Lewandowska, Jacek Ruminski, Tomasz Kocejko, and Jedrzej Nowak. Measuring pulse rate with webcam A non-contact method for evaluating cardiac activity. In 201 Federated Conference on Computer Science and Information Systems (FedCSIS), pp. 405-410, September 2011.
Xin Liu, Josh Fromm, Shwetak Patel, and Daniel McDuff. Multi-task temporal shift attention net- works for on-device contactless vitals measurement. arXiv preprint 20ti::2000.07990 2020.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin
transformer. arXiv preprint arXiv:2106. 13230, 2021.
Hao Lu, Hu Han, and Kevin Zhou. Dual-gan: Joint bvp and noise modeling for remote physiolog- ical measurement. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12404-12413, 2021.
J Martin Bland and DouglasG. Altman. STATISTICAL METHODS FOR ASSESSING AGREE- MENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT. The Lancet, 327 (8476):307-310, 1986.
Daniel McDuff, Sarah Gontarek, and Rosalind W. Picard. Remote Detection of Photoplethysmo- graphic Systolic and Diastolic Peaks Using a Digital Camera. IEEE Transactions Biomedical Engineering, 61(1)):9948-2955 December 2014.
Daniel McDuff, Javier Hernandez, Erroll Wood, Xin Liu, and Tadas Baltrusaitis. Advancing non- contact vital sign measurement using synthetic avatars. arXiv preprint arXiv:2010. 12949, 2020.
D. Nister, O. Naroditsky, and J. Bergen. Visual odometry. In Proceedings of the 2004 IEEE Com- puter Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., vol- 1, pp. I-I, June 2004. ISSN: 1063-6919.
Tania Pereira, Nate Tran, Kais Gadhoumi, Michele M. Pelter, Duc H. Do Randall Lee, Rene Colorado, Karl Meisel, and Xiao Hu. Photoplethysmograply based atrial fibrillation detection: review. npj Digital Medicine, 3(1):1-12, January 2020.
Ming -hher Poh. Daniel J McDuff, and Rosalind W Picard. Advancements in noncontact, multi- parameter physiological measurements using webcam. IEEE transactions on biomedical engi- neering, 58(1 58(1):7- 11,11111 2010a.
12

---

Under review as a conference paper at ICLR 2022
Ming-Zher -her Poh, Daniel J. McDuff, and Rosalind W. Picard. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Optics Express, 18(10):1072- 10774, May 2010b. Publisher: Optical Society of America.
Andrew Reisner, Phillip A. Shaltis, Devin McCombie, H Harry Asada, David S. Warner, and Mark A. Warner. Utility of the Photoplethysmogram in Circulatory Monitoring. Anesthesiol- ogy, 108(5):950-99) 958, May 2008.
Honnesh Rohmetra, Navaneeth Raghunath, Pratik Narang, Vinay Chamola, Mohsen Guizani, and Naga Rajiv Lakkaniga. Al-enabled remote monitoring of vital signs for COVID-19: methods, prospects and challenges. Computing, March 2021.
Rencheng Song, Huan Chen, Juan Cheng, Chang Li, Yu Liu, and Xun Chen. Pulsegan: Learning to generate realistic pulse waveforms remote photopltthysmogapph IEEE Journal of Biomedical and Health Informatics, 25(5):177-1333) 2021.
Ronny Stricker, Steffen Müller, and
Gross. Non-contact video- based pulse rate mea- surement on mobile service robot. In The 23rd IEEE International Symposium on Robot and Human Interactive Communication pp.
August 2014. ISSN: 1944-9437.
Chihiro Takano and Yuji Ohta. Heart rate measurement based on time-lapse image. Medical engineering & physics, 29(8):853-855 857, 2007.
K Takazawa. Clinical usefulness of the second derivative of plethysmogram (acceleration plethys- mogram). J Cardiol, 23:207-217, 1993.
Kenji Takazawa, Nobuhiro Tanaka, Masami Fujita, Osamu Matsuoka, Tokuyu Saiki, Masaru Aikawa, Sinobu Tamura, and Chiharu Ihukiyama. Assessment of Vasoactive Agents and Vas- cular Aging by the Second Derivative of Photoplethysmogram Waveform. Hypertension, 32(2): 365-370, August 1998.
Shoichiro Takeda, Kazuki Okami, Dan Mikami, Megumi Isogai. and Hideaki Kimata. Jerk-aware video acceleration magnification. In Proceedings of the IEEE Conference on Computer Vision and Pattem Recognition, pp. 1769-1777, 2018.
Wim Verkruysse, Lars Svaasand, and Stuart Nelson. Remote plethysmographic imaging using ambient light. Optics express, 16(26):21434-22 21445, 2008
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nel- son, Eric Jones, Robert Kern, Eric Larson, C. J. Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro. Fabian Pedregosa, and Paul van Mul- bregt. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nature Methods, 17 (3):261-272, March 2020. Number: 3 Publisher: Nature Publishing Group.
Wenjin Wang, Albertus C den Brinker, Sander Stuijk. and Gerard De Haan. Algorithmic principles of remote ppg. IEEE Transactions on Biomedical Engineering. 64(7):1479-149) 2016.
Wenjin Wang, Albertus C. den Brinker, Sander Stuijk, and Gerard de Haan. Algorithmic Principles of Remote PPG. IEEE Transactions on Biomedical Engineering, 64(7):1479-1491, July 2017. Conference Name: IEEE Transactions on Biomedical Engineering.
Hao-Yu Yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frédo Durand, and William Freeman. Eulerian video magnification for revealing subtle changes in the world. ACM transactions graphics (TOG), 1-): 2012.
Ting Wu, Vladimir Blazek, and Hans Juergen Schmitt. Photoplethysmography imaging: new noninvasive and noncontact method for mapping of the dermal perfusion changes. In Optical Techniques and Instrumentation for the Measurement of Blood Composition, Structure, and Dy- namics, volume 4163, pp. 62-70. International Society for Optics and Photonics, 2000.
Yichao Zhang, Silvia L Pintea, and Jan C Van Gemert. Video acceleration magnification. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 529-537, 2017.
13

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 SUPPLEMENTAL METHODS
A.1.1 EXAMPLE VIDEO FRAMES
Figure 5: Example video frames of different synthetic avatars generated for the training data set. The eeghy--parmeeterizedd avatar generation pipeline enables the creation of diverse subjects with varied demographics, lighting conditions, backgrounds, clothing/accessories,, and movements.
A.1.2 MODEL ARCHITECTURE
The first two 3D convolutional layers in each branch each have 16 filters and the final two 3D convolutional layers in each branch each have 32 filters. Each convolutional layer has filter size of 3x3x3 for all 3D convolutional layers in the network. All convolutional layers are padded such that they have the same height, width, and number of time steps in each consecutive layer. Convolutional layers use the hyperbolic tangent activation function, except for the convolutional layers used for the attention masks which masks (one per time step) are applied by applying an element-wise multiplication of the attention mask with each 3D convolutional feature map. Average pooling layers reduce the height and width of the frames by a factor of two, except for the final average pooling layer that pools over the entire frame (i.e. reduces each feature map to a single value per time step). Dropout (25% probability) applied after every pooling layer to reduce overfitting.
sigmoid activation function for generating the soft masks. Attention
After the final pooling layer, the learned features for each time step in a branch are concatenated together (i.e. combined across branches share information). Each target signal uses its own set of (2) RNN layers to read the concatenated features over time and generate target sequence. The first RNN layer is implemented as a bi-directional GRU (hyperbolic tangent activation function) with 64 total units (32 each direction). The second RNN layer is a GRU (linear activation function) layer with output value per time step.
14

---

Under review as a conference paper at ICLR 2022
Figure 6: Example video frames from four participants in the AFRL dataset used for model testing and evaluation.
A.1.3 METRIC CALCULATION
Heart Rate (HR) estimation To estimate the heart rate, we use fast Fourier transform (FFT)- based method to calculate the dominant frequency in the signal, which corresponds to the heart rate. We first estimate power spectral density using the "periodogram" function from the scipy.signal (Vir- tanen et al. 2020) library. Then we band-pass filter the PPG signal, with cutoff frequencies 0.75- 4.0 Hz (corresponding
minimum HR of 45 BPM and maximum HR of 240 BPM). Finally, select the frequency with the maximum power, and use this as our estimated HR.
Left Ventricle Ejection Time (LVET) estimation The LVET time is defined the time interval between the diastolic peak and the dicrotic notch. To calculate this interval, we first identified the diastolic point in the second derivative (SD) of the PPG signal, which, because minima in the PPG heartbeat, appears as a "global" maxima (positive SD value) in the SD PPG. Then, in each predicted SD PPG waveform, dicrotic notch manifests as a "local" minima in the PPG signal, it appears as a "local" maxima in the PPG SD signal (positive SD value). Using peak detection ("find peaks" function the scipy.signal library (Virtanen et al.] 2020)) we identify candiadate dicrotic notch points by finding local peaks that occur after diastolic point, and use the dicrotic notch candidate point that closest in time to the reference diastolic point.
"global"
identified candidate dicrotic notch points. Since the
Because both the ground truth PPG (and therefore its derivatives) and, in particular, the predicted PPG (and its derivatives), contain signal artifacts and noise, the peak detection process is not perfect. To reduce variability in the LVET interval estimates due to noise, we apply a smoothing operation. Specifically, we estimate the mean LVET interval within 10-second oveellarlg window and use this as our estimate of true/predicted LVET. See Appendix Fig. 7 for example LVET intervals over time, and the estimated LVET intervals after smoothing within windows.
15

---

Under review as a conference paper at ICLR 2022
A.2 SUPPLEMENTAL RESULTS
Table 2: Quantitative performance comparison between different architecture configurations. Values shown (mean + standard deviation).
Beats-per-minute (BPM); First Derivative (FD); Heart Rate (HR); Mean Absolute Error (MAE);
Second Derivative (SD); Left Ventricle Ejection Time (LVET).
Input Frames
Target Signals
HR MAE (BPM)
LVET MAE (ms)
FD
SD
FD
SD
3.14 + 7.07
83.09 + 42.41
/
/
6.97 + 16.30
65.10 + 31.56
/
2.95 6.57
57.67 + 25.82
True LVET
FD-optimized Prediction
SD-optimized Prediction
600
LVET (ms)
400
200
0
60
120
180
240
Time (seconds)
Figure 7: Comparison of Left Ventricle Fjection Time (LVET) estimation over a 5-minute time period. Solid lines are computed the mean LVET interval within non-overlapping 10-second windows.
Table Quantitative cross-dataset performance comparison between different architecture config- urations in two separate external datasets: UBFC (Bobbia al. 2019) and PURE (Stricker al. 2014]. Models were trained only on the synthetic dataset. Descriptions of the FD-optimized and SDDoptimized models can be found in Table I Values shown are (mean + standard deviation). Beats minttee (BPM); First Derivative (FD); Heart Rate (HR); Mean Absolute Error (MAE); Second Derivative (SD); Left Ventricle Ejection Time (LVET).
Model
UBFC
PURE
HR MAE (BPM)
LVET MAE (ms)
HR MAE (BPM)
LVET MAE (ms)
(FD-Optimized)
3.59 + 9.84
80.90 58.99
4.17 + 2.99
75.84 + 18.74
(SD-Optimize))
3.88 + 9.85
41.90 + 15.72
4.18 + 3.26
37.85 + 3.57
16


Reviews:
[{"review_id": "wu2w5NH-QzD", "reviewer": "Reviewer_qH24", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n-This is an interesting application. It is somewhat surprising how much physiological information can be extracted from simple video. This in turn brings up interesting privacy & ethical concerns which the authors briefly discuss in Section 7. I think quite a few researchers will find it interesting that so much physiological information can be extracted from a simple video of someone\n-The paper is for the most part well written\n-Source code will be provided. However the dataset cannot be released\n\nWeaknesses\n-Algorithmically the work is not really novel. It is a rather vanilla neural network trained on a somewhat novel dataset.\n-Section 3 feels a bit detached from the rest of the paper. As far as I can tell this does not contribute to the paper's main argument. \n-In section 1 the authors can provide a better description of what PPG is. Perhaps provide there a figure illustrating what type of graph data is typically provided. They may also want to consider providing up-front a diagram describing their system\n-The authors are using 30fps video. How reliable are the second derivatives in such a case? If this was a high frame rate camera would they expect better results? How do intrinsic camera parameters like autogain & shutter speed affect the system reliability? How does environment illumination affect system reliability? Do the subjects need to be in a controlled environment for this to work? A discussion on this would be helpful", "summary_of_the_paper": "This is an application paper providing an algorithm for estimating the left ventricle ejection time interval (LVET), which is the time for heart systole to occur. This metric has various clinical applications. The authors extract this metric using Photoplethysmography (PPG) labels as training labels, and video of faces as input. The authors use a DNN to extract from the images subtle changes in skin color and thus estimate blood flow changes that enable them to estimate the LVET. The authors argue that by extracting second order derivatives in the trained DNN they are able to obtain improved results ", "summary_of_the_review": "Overall as indicated above technically the algorithmic contributions are rather modest. The main novelty lies in the application described. If the paper is to be accepted it should be accepted due to the interesting application.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aAyqtRU9tt0", "reviewer": "Reviewer_eB6A", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\nThe investigation about higher-order of dynamics in remote cardiac measurement is inspiring. The discussion about the trade-off of low and high order dynamics indeed gives insights to future work.\n\nAs called by this paper, instead of predicting the heartbeat rate, the community should pay more attention on measuring the detailed waveform that benefits more clinical monitoring applications.\n\nWeaknesses:\n\nThe novelty of the proposed multi-derivative architecture is limited since it is an updated version of the basic network proposed in (Chen & McDuff ,2018) with minor modifications. The 3D convolution and final RNN is not new as they have been used in other rPPG estimation works.\n\nSince the low and high order dynamics can be a general choice for rPPG estimation, the experiments should also include other popular rPPG estimation networks.\n\nIntuitively, the high order dynamics can also be sensitive to motion interferences, such as facial motion or facial expression. It is not clear what is the degree of the motion and illumination constraints of the physical model (Eq. 5).\nThe unconscious facial motion and expression are inevitable in practice. Do we need heavy pre-processing that strictly align the face region for each frame? What are the pre-processing steps in this work?\nIs this one of the reasons that the network is trained on a synthetic dataset?\n\nAlso, from my understanding, the cardiac measurement is based on skin color variation, which is a kind of low-level feature. It is not clear why we need a large and diverse set of subjects to train the network.\n\nThe experiment should also include popular publicly available datasets such as PURE and UBFC-rPPG. \nIt is also expected to compare the proposed method with the state-of-the-art on these datasets.\n", "summary_of_the_paper": "This paper investigates the effectiveness of the higher-order (second derivatives)\nin the context of remote cardiac measurement. A multi-derivative architecture is designed based on the basic network of previous work (Chen & McDuff ,2018). The model is trained on the synthetic dataset (McDuff et al., 2020) and test on AFRL dataset (Estepp et al., 2014) with real person. Experimental results illustrate the effectiveness of second-order dynamics on measuring detailed cardiac signal and indicate the trade-off between low and high order dynamics of LVET error and HR error.", "summary_of_the_review": "Considering the limited novelty of the proposed network and inadequate experiments, I think this paper does not meet the standard of this conference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6HHw72-lVvb", "reviewer": "Reviewer_me2S", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is interesting as the subject is a hot topic in the community, and the development of contactless monitoring solutions is seen as key for multiple applications, from in-hospital (pediatric ICU, patients under dialysis) or remote medical appointment (especially during the COVD pandemic)\nThe authors have evaluated their technique with relevant tests, such as estimation of HR and LVET. Such an evaluation technique is important, as biomedical applications need to be assessed on their final goal and authors did will not to look at MSE of estimated PPG or derivatives, but to look at the accuracy of the estimated clinical features.\nI have however several concerns regarding the submission:\nAs denoted by the authors, the use of normalized differences for the prediction of first derivative PPG has already been suggested, the innovative aspect of the paper is therefore quite limited.\nThe authors do not compare their approach with other techniques from the state-of-the-art, it is therefore hard to evaluate the performance of the suggested technique.\nHow would the technique cope with pathological rhythms, which affect the quality and/or morphology of the PPG?\nThe pre-processing of the videos consists in rescaling the images into 36X36 pixels. Do the authors believe that such a low resolution is enough for real-life applications?\nThe use of artificial data is often key for the development of machine-learning techniques in the medical field due to the scarcity of real data. I have however some doubts about the realism of the training data, do the authors believe that avatar and generated videos are realistic enough? Does the addition of the physiological variation onto the avatar skin really reflect real and noisy data? \nGiven the final results, it is unclear which approach the authors are suggesting? Do they believe a specific network should be trained for each application (HR, LVET, other )\n", "summary_of_the_paper": "This paper describes a novel approach of extracting information from contact-less (video-based) cardiac monitoring. The authors highlight the potential of analyzing second-derivatives of the PPG signals, especially for the estimation of Left Ventricle Ejection Time intervals.\nThe authors propose a deep learning approach, with multiple inputs (video, but also first and second derivatives and also a multi-task approach (first and/or second derivatives of the contact PPG) \n", "summary_of_the_review": "I am not sure that the paper is innovative enough for acceptance to ICLR. The authors should also compare their approach with other state-of-the-art approaches, and evaluate on difficult real-life cases (pathological rhythms, different skin colors,…)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4RRioCNvL_A", "reviewer": "Reviewer_W7ES", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n-Thorough ablation study\n\n-Considering second-order motion is a novel topic\n\nWeakness:\n\n-Technical novelty seems to be limited. The convolutional attention network (CAN) paradigm was introduced before and it will be better to have some discussion on what is the new component added by comparison with the original CAN.\n\n-Further discussion is required for some results. More discussion is appreciated for the results in Table 1. Intuitively, the first-order and second-order motion is highly correlated. If the first-order performance improves, the second-order performance should also improve and vice versa. On the other hand, doing both first and second-order estimation could be viewed as multi-task learning and they should benefit from each other. However, it shows that for this task, removing first-order loss will improve second-order performance. Also, if add both losses, the second-order performance decreased. This is a kind of strange phenomenon and more discussion is appreciated. \n\n-Lack some references in PPG/cardiac motion estimation\nThe strange results might be because the overfitting or generalization issues. The meta-learning has been proven to be useful for PPG estimation/cardiac motion estimation tasks. It will be interesting to add the following references and have some discussions:\n\nLee, Eugene, Evan Chen, and Chen-Yi Lee. \"Meta-rppg: Remote heart rate estimation using a transductive meta-learner.\" European Conference on Computer Vision. Springer, Cham, 2020.\n\nYu, Hanchao, et al. \"Foal: Fast online adaptive learning for cardiac motion estimation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.", "summary_of_the_paper": "Estimating the motion signal from the video is an important task with applications in computer vision and healthcare. In this paper, a multi-derivative convolutional attention network is used to estimate the high order derivatives of the 1D heart beating signals, with video data as input. A second-order loss is used and the results are evaluated in first-order(HR MAE) metrics and second-order metrics (LEFT MAE). The results show the second-order loss will improve the second-order metrics while harming the first-order performance.", "summary_of_the_review": "In summary, the higher-order motion characteristics estimation from the video is an interesting topic while I have some concerns on the novelty and the experiment results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-WvAhrGyaSu", "reviewer": "Reviewer_Z3zg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong Points:\n\nThe paper is clear and very well written. In addition, the problem is clearly stated, and the methodology is presented with the corresponding metrics.\n\nThis work has a strong background motivation and the methodology is novel, since there is currently a need to improve the models whose tasks are represented in higher dynamics, as shown in the introduction.\n\nThe paper evaluates correctly the incorporation of the second derivative of the input frames to improve the performance of second order dynamics, which enriches the results of the work.\n\nIt is very important and beneficial that the authors have shared their work on Github, so that the work is reproducible and replicable. This enriches the work presented.\n\n\nWeak Points:\n\nThere is a lack of clear support for the use of a synthetic dataset instead of a real one. There are differences that need to be clearly defined and justified as to how much it affects the performance of the convolutional network.\n\nAlthough the use of the attention mask is a reasonable assumption, there is a lack of comparative results of the full model with and without the attention mask. Obtaining these differences is important to understand how it affects the use of care models. The comparison is not complicated and is suggested.\n\nThe proposed architecture is interesting and works well, however there is a lack of comparison with other architectures to evidence that the proposed architecture is the best. In this way, a comparison with simpler and more complex models could be shown to verify that the implemented model is the best one.\n\nThe paper mentions that in the implementation the video resolution is reduced to 36x36 pixels. But, it is not supported how the reduction of resolution affects performance. One way to present this could be through a comparison of performance, computational power consumption, and response time at different resolutions.\n\nThe number of epochs used in the training stage is eight, which is relatively low. In addition, there is no support as to why this low number of epochs was chosen. Making use of more epochs would help a model to generalize better.\n\n\nSuggestions:\n\nIn the Introduction section, you mentioned that \"To date, in video-based PPG measurement the primary focus of analysis and evaluation has been on features extracted from the raw waveform or its first derivative\", however, no research works are cited or referenced. The references that support this statement should be cited.\n\nAt section 4.2, the paper states: “Given that difference frames X’(t) are useful for predicting the first derivative PPG waveforms, features learned from the difference-of-difference frames X’’(t) may be beneficial for predicting the second derivative PPG signal.”  I think I should say: “Given that difference frames X(t) are useful for predicting the first derivative PPG waveforms, features learned from the difference-of-difference frames X’(t) may be beneficial for predicting the second derivative PPG signal.”\n\nHow much does the amount of light affect the prediction? It would be interesting to show a graph or table of the variation of results as the illumination is increased or decreased. Also, would it be better to perform the prediction on natural light? How would artificial light vary the results (e.g., light operating at a frequency of 60 Hz changes the illumination and the camera could capture at the moment of less illumination which would change the results)?\n", "summary_of_the_paper": "Computer vision algorithms based on deep learning usually optimize first-order dynamics. But, in cases the properties of interest are small variations, which are best described in higher order, such as acceleration. A direct application of this is the heart pulse, where the second derivative can be used as an indicator of blood pressure and arterial disease. In this work, the author(s) propose a methodology for beat waveform prediction using convolutional attention networks, and  considering the derivatives of the input images and optimizing for derivatives of interest. The results are very interesting and show that by appropriately incorporating higher-order dynamics, the performance of video understanding tasks can be greatly improved.", "summary_of_the_review": "The work presented is of general interest and contributes with a methodology for a better understanding of learning representations for dynamical systems. The paper addresses several aspects such as the optimization of the loss function of neural network models to better estimate higher order dynamics, as well as the incorporation of the second derivative of the input frames to improve the performance of second order dynamics. Applying this to the beat wave characterization problem, the model is able to better estimate left ventricular ejection time intervals (LVET). The paper will be greatly improved if the revisions suggested in this review are taken into account, especially improving the comparative results with other models and better justifying the use of synthetic data.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has received 5 reviews with 4 advocating for rejection (marginal or clear cut) and one borderline leaning towards a weak accept. The key concerns voiced by the reviewers are the lack of novelty (*the novelty of the proposed multi-derivative architecture is limited*), the lack of comparisons with specific architectures in appropriate setting (rPPGNet without the STVEN module, DeeprPPG, RhythmNet, CVD), and concerns about the use of synthetic data (although authors provide some justifications to that end). It appears that the key to reviewers' scores is that higher-order dynamics did not constitute a sufficient novelty.

Given the post-rebuttal scores and discussions, AC has no option but to recommend a reject at this point.
Decision: Reject