The Benefit of Distraction: Denoising Remote Vitals Measurements Using Inverse Attention
Authors: Ewa Magdalena Nowara
Daniel McDuff
Ashok Veeraraghavan
Venue:  (2021)
Keywords: convolutional attention networks
denoising
computer vision
camera-based physiology

Abstract: Attention is a powerful concept in computer vision. End-to-end networks that learn to focus selectively on regions of an image or video often perform strongly. However, other image regions, while not necessarily containing the signal of interest, may contain useful context. We present an approach that exploits the idea that statistics of noise may be shared between the regions that contain the signal of interest and those that do not. Our technique uses the inverse of an attention mask to generate a noise estimate that is then used to denoise temporal observations. We apply this to the task of camera-based physiological measurement. A convolutional attention network is used to learn which regions of a video contain the physiological signal and generate a preliminary estimate. A noise estimate is obtained by using the pixel intensities in the inverse regions of the learned attention mask, this in turn is used to refine the estimate of the physiological signal. We perform experiments on two large benchmark datasets and show that this approach produces state-of-the-art results, increasing the signal-to-noise ratio by up to 5.8 dB, reducing heart rate and breathing rate estimation error by as much as 30%, recovering subtle pulse waveform dynamics, and generalizing from RGB to NIR videos without retraining.

Under review as a conference paper at ICLR 2021
THE BENEFIT OF DISTRACTION: DENOISING REMOTE VITALS MEASUREMENTS USING INVERSE ATTENTION
Anonymous authors
Paper under double-blind review
ABSTRACT
Attention powerful concept in computer vision. End-to-end networks that learn focus selectively regions of image video often perform strongly. However, other image regions, while not necessarily containing the signal of in- terest, may contain useful context. We present an approach that exploits the idea that statistics of noise may be shared between the regions that contain the signal of interest and those that do not. Our technique uses the inverse of an attention mask to generate a noise estimate that is then used to denoise temporal observa- tions. We apply this to the task of camera-based physiological measurement. convolutional attention network is used to learn which regions of video contain the physiological signal and generate a preliminary estimate. A noise estimate is obtained by using the pixel intensities in the inverse regions of the learned atten- tion mask, this in turn is used to refine the estimate of the physiological signal. We perform experiments proach produces state-of-the-an results, increasing the signal-to-nois ratio by up to 5.8 dB, reducing heart rate and breathing rate estimation error by much 30%, recovering subtle pulse waveform dynamics, and generalizing from RGB to NIR videos without retraining.
two large benchmark datasets and show that this ap-
INTRODUCTION
Attention mechanisms have been successfully applied in many areas of machine learning and com- puter vision (Mnih et al., 2014 Vaswani et al. (017)), including object detection (Oliva et al. 2003).
activity recognition (Sharma al. 2015), language tasks (Anderson et al.) 2018 You et al. machine translation (Bahdanau et al. (214)) and samera-based physiological measurement (Chen & McDuff 2018). An additional benefit of attention mechanisms that they are interpretable and show which regions of an image were used to generate particular output. In this paper, we focus question
2016
on a counter-intuiife typically ignored by the attention models? And, can we exploit information in this region to improve the quality of estimation for the underlying signals of interest?
there important information contained within the regions that are
We focus on the specific temporal prediction problem of camera-based physiological measurement as an exemplar application for our approach. The SARS-CoV-2 (COVID-19) pandemic has rapidly changed the face of healthcare, emphasizing the need for better technology remotely provide care to patients. COVID-19 19 is linked to serious heart and respiration related symptoms (Xu et al.) 2020 ZZhng al. 2020 Puntmann al. 2020). Even after the COVID-19 crisis, many doctor appointments could be carried out online with telemedicine technology, increasing the flexibility for appointments. Recent research in computer vision has led to the development of non-contact physi- ological measurement techniques that leverage cameras and computer vision algorithms (Takano & Ohta 2007 Verkruysse al. 2008 Poh al. 2010aa De Haan & Jeanne 2013 Wang et al. 20177 Chen & McDuff) 2018). Camera-based vital signs could also enable driver monitoring (Nowara al. 2018), face anti-spoofing (Liu et al. 2020al Nowara et al.) 2017). or long-term human- computer-interacton (HCI) studies (McDuff et al. 2016) where wearing contact devices for ex- tended periods may be infeasible. Convolutional networks currently provide state-of-the-art perfor- mance on heart rate (HR) and breathing rate (BR) measurement from video (Chen & McDuff 2018
Yu et al. 2019 Liu et al. 2020b).
While the convolutional neural networks may be able to accurately learn what features in the image are important for finding the physiological signals, they may not be able to learn a good model the noise that corrupts the signals. The noise present in the video, which is considered to be "everything else than the signal of interest", may be caused by many diverse factors and could vary greatly across videos and datasets. Possible sources of noise include changes in head motion (Estepp al. 2014).

---

Under review as a conference paper at ICLR 2021
Skin Pixel Trace
Hair Pixel Trace
Background Pixel Trace
Norm. Amp.
Norm. Amp.
Norm. Amp.
Time
Time
Time
Attention Mask
Skin pixels contain the strongest pulsatile signal, typical attention network will learn focus on these pixels. However, they also capture motion and lighting information.
Inv. Attn.
We propose to use an inverse attention mask capture motion and lighting changes as noise estimate and to use these to improve the pulse SNR and heart rate estimates.
Recovered Pulse
Mask
Norm. Amp.
Time
Figure I: We propose an approach, using the regions ignored by the attention mechanism (e.g.. hair or background), as noise estimates. We train model to learn a denoising mapping to remove these noise sources from the physiological signal. Our approach produces more accurate physiological waveforms, even in severely challenging scenarios.
facial expressions (Zhng et al. 2016), speech, ambient light variations (Nowara al. 2018), 8), and video compression artifacts Yu et al. 2019 Nowara McDuff 2019). The wide variety of possible noise sources makes it challenging for any model to explicitly capture good noise representation and to remove that noise from the signals of interest.
The key observation make is that gions ignored by an attention mechanism in neural model likely contain informa- tion about sources of noise that are also present in the regions used by the atten- tion mechanism to compute the physiolog- ical signals. Using the "distraction" re- gions that were ignored by the attention masks offers way to estimate the noise for each video without making any as- sumptions about the nature of the noise. The only assumption is that most regions not used by the attention masks do not contain the signals of interest and conse- quently contain noise.
Background Region
Background Region
4
Plodl Aug. Fioe
Rgg.
60
100
120
60
70
100
110
120
Hair Region
Ang. Ploel Velure
Hair Region
120
50
Skin Region
Region
Fae
We demonstrate that tensity variations from regions outside of the attention mask and learn a denoising mapping to remove noise from the recovered signals. See Fig. T for an overview of our denoising ap- proach. F shows additional examples of pixel traces from different regions in the frame, demonstrating that the regions ig- nored by the attention masks often contain noise correlated with the noise in the se- lected skin regions. We show that our ap- proach outperforms state-of-the-e ods several datasets across range of HR and BR enror measures. Our denoising approach also generalizes well data, data recorded with different imaging modalities,
60
100
120
90
100
20
120
use the in-
Timets)
Timels)
Avg. Picl Vake
Skin Region (Zoomed In)
Skin Region (Zoomed In)
noise estimate
83.5 TimeEs)
83
91.5
92.5 Timafa)
Figure 2: Temporal changes of background (gray), hair (green) and skin (red) pixel intensities in video correlated due to large change (e.g., motion, shadows). Physiological signals
very subtle (see zoomed portion). This motivates us to use the inverse attention region denoise the physiological estimates.
meth-
2

---

Under review as a conference paper at ICLR 2021
RGB Video (AFRL Dataset)
Near Infra-Red Video (MR-NIRP Dataset)
Frome
Freim
Current staae--ff of the-hrrrrrr (CAN)
Current state the (CAN)
Normaited Amplude
Tima (s)
Time 64)
Dincttic Notch
Amplt.os
1 1urs
Nomalted Ampitate
OOrr
Diestolii Peoks
(5)
Time (3)
Black Sensor Measurement
Camera- Based Estimate
Figure 3: Pulse signals output by a state-of-the-att network and our denoising method. Our method produces cleaner signals, free from motion artifacts (present in the benchmark method), and better matching the ground truth subtle dynamics and shape. Notice the zoomed-in portions with easily identifiable dicrotic notch and diastolic peaks in our outputs.
such near-infrared (NIR), without any additional training. Our proposed approach is also able to recover very subtle waveform dynamics, such as the clearly visible dicrotic notch, shown in Fig, 3 which is challenging for video-based methods. Obtaining clean and more accurate waveforms is use- ful for determining important health metrics, such as blood pressure (Elgendi et al. 2019). which is infeasible with current methods. The idea of using the inverse attention regions is likely very useful in wide variety of vision tasks, ranging from activity recognition to deblurring. However, in this work, we focus on physiological measurement due to the clinical importance.
The core contributions of this paper are to: (1) propose the use of inverse attention masks for gen- crating noise estimates, (2) present a novel method for denoising non-contact physiological mea- surement using this approach, (3) evaluate our method three datasets showing state-of-the- performance on pulse and respiration measurement, (4) demonstrate that our approach generalizes to NIR data without further training. Supplementary material including code, models, video exam- ples and additional experimental results are provided with this submission]
RELATED WORK
Attention Mechanisms. Attention mechanisms provide way for model to learn which parts of an image or video "are relevant for the task at hand and attach a higher importance to them" " (Sharma ct al. 2015). During training the attention weights learned reflecting the importance of the embedding features. Recently, transformer models, based solely on attention mechanisms, have become popular (Vaswani al. 2017). In convolutional neural networks (CNNs) these attention mechanisms typically form a spatial mask. These masks can help practitioners understand the decision-making process of network (Fukui al. 2019) and in certain cases the "fixations" of attention generated by computer models and by human observers were very similar (Oliva et al., 2003 Attention mechanisms can be used to connect layers: for example, one which focuses on temporal information (e.g., trained on flows) and another which focuses spatial information (c.g., trained on RGB frames). Prior work has found that these crosslink layers guide the spatial-stream to pay more attention to the human foreground areas and can be less affected by background clut- ter (Tran& & Cheong 2017). In physiological mcasurement, two-layer networks have been found to be effective as both color and motion information are valuable for extracting the subtle physiological signal in the presence of noise (Chen & McDuff 2018). While attention mechanisms often work well, they are simple representation of which regions important. However, pixels outside these regions may provide useful context or a strong prior about the noise present.
Physiological Imaging. Volumetric changes in blood over time lead to subtle changes in light reflected from the skin and subtle motion variations which can be measured with a camera (Takano

---

Under review as a conference paper at ICLR 2021
Frames
Frames
Asto, Mesk
Output- Waveform
CAN Output- Power Spec.
Mack
CAN Output Waveform
Output Power Spec.
horm, Anp.
Inverse Merk
Noise Prior- Waweform
Noise Priur Power Spee.
Livers Mage
Norm. Anp.
Noise Prior- Wavetorm
Noise Prior- Power Spec.
An Mnll m
WW W MAN
Ours Waveform
Power Spec.
New. AnS. Norm Ann.
Ours- Weveform
Ours Power Spec.
Ground Truth Wavetorm
Ground Truth Power Spec.
Ground Truth Wavetorm
Ground Power
Free
10nc
frod
Figure 4: Examples of attention masks, inverse attention masks (yellow=higher weight) and the corresponding original physiological estimates, the noise estimates (shown for the green channel only for visual clarity), and denoised physiological signals from AFRL (left) and MMSE-HR (right) datasets. Green lines indicate the ground truth HR.
& Ohta 2007:
Verkruysse et al., 2008). The physiological signal obtained from video can be used to recover several metrics and vital signs, including heart rate (Poh et al., 2010a), heart rate variability (Poh et al.
2010b), breathing rate (Poh et al. 2010b). blood oxygenation (Tarassenko jet al. 2014) and pulse transit time (Shao et al. 2014). NIR (Nowara et al. 2018 Chen et al.) 2018) and thermal cameras have also been successfully used for measuring physiological signals in the dark (Garbey et al. 2007] Pavlidis et al.) 2016). Unfortunately, the signals of interest in video- based physiological measurement are often very subtle and can be easily corrupted by noise due to body motions and ambient lighting changes. Early work in physiological imaging used properties of the physiological signal, c.g., the periodic nature (Poh et al., 2010a) and hemoglobin absorption Wang et 2017) to recover the underlying physiological signal via de- de-mixing methods (Li et al. 2014; Macwan et al. 2019 2018 Tulyakov et al. 2016). Some of these unsupervised methods make simple assumptions that the pulse signal should be periodic (non-Gas Soon-Gaussian) and that any other source signals are noise (e.g., ICA). Others, such as POS, assume that the plane orthogonal to skin contains the pulsatile signal and non-orthogonal planes contain specular reflections and noise. Others have used physical skin models to learn a mapping from color changes (McDuff et al. 2018)) in these noise is not modeled explicitly. Recently, several groups have demonstrated that deep learning models free from heuristic assumptions about the signal structure can perform better, especially in presence of large motion and noise (Chen & McDuff 2018 Zhan
spectra (De Haan & Jeanne|[201
et al. 2019: Spetlikk et al.) 2018 McDuffi 2018 Niu et al. 2018a b Yu et al. 2019 Lee et al. 2020). J. These end-to-end methods do not explicitly define the noise but rather learn to recover the physiological signal in fully supervised manner. We show that the performance of a ofte-of-thhearr
model is significantly improved by using the distraction regions as an explicit noise estimate.
BENEFITING FROM DISTRACTION
Let us take a video of a person moving as an example. The skin pixels will contain information about the physiological signal but they will also capture the body motion, as in each frame the incident light changes with the orientation of the head (see Fig. [ In contrast, the hair pixels will not contain information about the physiological signal (as there are blood vessels in the hair) but will still contain information about the motion. In this section, we explain how we use those inverse attention (or "distraction") regions to denoise the physiological signals. The details of the proposed deep learning architecture are provided in Fig.55
3.1 PHYSIOLOGY AND NOISE ENCODER
The backbone of the encoder is formed using a convolutional attention network (CAN) (Chen & McDuff 2018). This contains appearance and motion branches learned jointly through an attention mechanism. The appearance model is trained directly on the input video frames. It learns from the color and texture information which regions in the video are likely contain strong physiological
4

---

Under review as a conference paper at ICLR 2021
Encoder
35x36
36x30
30:86
18x10
10x10
18x16
Denoising Model
Pulse:
Respiraton::
(Nomm. Frumme
Atserten
Naa
Video
Apprenaacco Input
Inverse Aantt CD 100 Na
Extrrate
Vidco
Figure 5: Proposed denoising architecture. The encoder provides the initial physiological signal and the noise estimates to the LSTM at each time step which outputs a denoised physiological signal.
signals. The motion model is trained on the difference of two consecutive video frames to differen- tiate between the intensity variations present in the video caused by the characteristic physiological variations from those from other sources. The attention mask then reflects heatmap of the strength of the pulsatile signal in each region of the frame. As shown in the first row of Fig, 4 the atten- tion masks mostly focus on skin regions known to have strong physiological signals, while ignoring other regions, such the eyes, hair, and background regions. The CAN normally outputs a single one-dimensional (1D) physiological signal estimate. However, we perform an element-wi wise mul- tiplication of the original input frame with the inverse of the attention mask weights compute secondary noise estimate.
We compute the noise signals at each time step by multiplying the inverse attention masks with each channel of each video frame in an element-wise manner. We then spatially average the resulting weighted pixel intensities obtain the noise estimate:
W Σ
No.1 HW
Σ x=1y=1
Luy. o Ma
where 7, and M, are the frame and mask at time t. Ne.t is the noise estimate from each [R, G, B] camera channel e time I, and H and W are the image height and width, respectively. The attention and the inverse attention masks were 34 34 pixels and the video frames were downsampled to the same size using bicubic interpolation.
We normalize the attention mask elements to range between 0 and 1. To obtain noise estimate, we set all values larger than a fixed threshold, T, to and everything else to 1, creating a binary found threshold of 0.1 worked well. This binary inverse
mask. Based the experiments attention mask ignores regions in the video initially used to compute the physiological signals and keeps all other regions. Examples of inverse attention masks and the corresponding noise estimates are shown the second row of Fig. |
3.2 DENOISING MODEL
Our denoising model is then formed as long short-term memory (LSTM) network with the encoder providing input at each time step. The goal is learn denoising function to further clean the physiological estimates. As input to the denoising LSTM we stacked the physiological signal and noise signal outputs generated by the encoder. The contact physiological signal (e.g., finger pulse oximeter) was used as ground truth for training. The noise estimates guide the LSTM learn which waveform features are related to noise and which are related to the physiological signal of interest. The LSTM learns suppress the noise from the physiological signal and outputs cleaner waveform matching the ground truth physiological signal better (see the third row of Fig. 4.. See the video in the supplementary material for more examples of noise estimates and denoised signals.
We used a two-layer bidirectional LSTM, with 128 hidden units, trained for 10 epochs with Adam optimizer (Kingma & Bal 2014) and MSE loss. Because the LSTM tends work better on shorter
5

---

Under review as a conference paper at ICLR 2021
sequences, split each video to sequences of 60 samples, with 50% overlap between time win- dows, which corresponded to two seconds for the 30 frames per second (fps) videos. Physiological datasets are often relatively small due to the complexity associated with collecting carefully syn- chronized physiological signals and high-quality videos. Therefore, we implemented the CAN and the denoising LSTM two separate networks to reduce the number of training parameters.
4 DATASETS
We evaluated our approach on the following two RGB video datasets and an NIR video dataset.
MR-NIRP MMSE-HR APRL
AFRL (Estepp al. 2014): 300 videos of 25 participants were recorded as 658 x 492 pixel images at 120 fps. Fingertip reflectance pho- toplethysmograms (PPG), electrocardiograms (ECG), and respiratory signals were recorded as ground truth signals. We used the ECG sig- nals to compute the HR estimation errors, the PPG signals to train the network for estimating HR, and respiratory signals for computing the errors and training the network for BR estima- tion. Each participant was recorded 12 times in each five minute experiment with varying motion and two different backgrounds. The participants: 1) sat still and rested their chin on a headrest, 2) sat still without the headrests, 3) moved their head horizontally at speed of 10 degrees/second, 4) 20 degrees/second, 5) 30 degrees/second, 6) reori- ented their head randomly once every second. We center-cropped the ARFL video frames to 492 x 492 pixels to remove the blank background areas.
Figure 6: Examples of images used to evaluate our proposed approach.
MMSE-HR (Zhang et al. 2016): 102 videos of 40 participants were recorded at 25 fps capturing 1040 1392 resolution images during spontaneous emotion elicitation experiments. Ground truth blood pressure (BP) wave was measured at 1000 fps and average HR updated after every heart beat. We used the blood pressure waves to train the network and the average HR to compute the HR estimation errors. 19 videos had erroneous average HR estimates, so we recomputed them by using the BP waveform. We detect peaks in the blood pressure waveform and compute the interbeat interval (IBI) between the detected peaks. Heart rate is estimated as μ(BBT where μ(IBI) is the mean IBI.
MR-NIRP (NIR) Nowara et al. 2018): Eight participants were recorded with a NIR camera. The videos were recorded at 640x 640 resolution and 30 fps. Fingertip transmission photoplethysmo- grams were recorded as ground truth signals. Each participant was recorded twice, once sitting still and once performing motion tasks involving talking and randomly moving the head. Because the background in MR-NIRP was not uniform, we applied face detection in the first video frame and cropped rectangular region with 110% width and height of the detected bounding box.
5 TRAINING DETAILS
Training the Encoder. Due to the large number of parameters we pretrain the encoder on the largest dataset (AFRL (Estepp et al., 20144) and lock the weights. When training the encoder the loss is calculated as the mean squared error between the physiological estimate and the ground truth. We do not compute the loss using the noise estimate as there is no ground truth noise signal. In our experiments we performed training and testing separately for each of six motion tasks from the AFRL dataset with a participant- -nndepaan--ideeeddeen cross- volss-val,,,,nn, leaving out 20% of the participants in each validation split. For experiments on the MMSE-HR and MR-NIRP datasets we used the trained model from Task as these contained the most similar amplitude motions. maximize the diversity of the participants that this model was trained to improve the generalizability to new datasets, we instead used a subject-dependent cross vross-validation, using four minutes of each video for training and one minute for testing.
Training the Denoising Model. When evaluating on the AFRL dataset we trained the denoising model with the same subject-independcee dataset has fewer videos than the AFRL dataset; therefore, we used a leave-onee sane--ne--uuject--on cross- validation where left out all videos of one subject and trained the model on all remaining videos,
procedure for the encoder on AFRL. The MMSE-HI HR

---

Under review as a conference paper at ICLR 2021
repeating this for each subject. The MR-NIRP dataset was small and not suited for training the networks, so we used the LSTM trained on the AFRL dataset. This allowed us to test cross-dataset generalization.
We bandpass filtered ([0.7 Hz, 2.5 Hz]) and detrended the signals Tarvainen et al. 2002). We nor- malized the signals by subtracting the temporal mean, dividing by the temporal standard deviation in each video, and normalized their amplitudes to -1 and 1. We resampled all sequences to 30 fps. The signals from each video were divided into 30-second non-overlapping windows. We evalu- ated the performance of our proposed denoising approach using mean absolute error (MAE), root mean square error (RMSE). Pearson's 's correlation coefficient (p) between the estimated HR and the ground truth HR, SNR of the estimated physiological signals (De Haan & Jeanne, (013)), and wave- form mean absolute error (WMAF) computed between the estimated and the ground truth signal. See the supplementary material for the definitions of the error metrics.
RESULTS AND DISCUSSION
We compare four variants of our proposed approach to four state-of the--hhrrr art methods for recovering the pulse signal (Chen & McDuff 2018 Poh et al. 2011 De Haan & Jeanne, 2013 Wang et al.,
2017) and two methods for recovering the breathing signal (Chen & McDuff) 2018 Tarassenko et al. 2014) (see the supplementary material for implementation and signal pre-processing details). The variants of our approach we compare are: training our model with noise estimates as input ("Distraction") and without noise estimates as input ("No Noise"). We can also directly subtract the noise estimate from the signal estimate either in the time domain "Wave. Sub.", or compute the power spectrum of the estimated noise and signal and subtract the noise spectrum from the signal spectrum "Freq. Sub.".
Our method achieves lower HR MAE, RMSE and waveform MAE and higher HR correlation (p) and SNR (see Tables [T] compared to previous approaches on two large datasets. On the AFRL dataset the MAE is reduced from 2.93 beats per minute (BPM) to 2.25 BPM (25% reduction in error), and on the MMSE-HR dataset the MAE is reduced from 3.74 BPM to 2.50 BPM (33% reduction in error). This shows that information excluded by the attention mask can be successfully leveraged to remove noise, leading to substantial improvements in signal qual- ity. Moreover, the proposed denoising approach is able to recover the subtle waveform dynamics, reducing the waveform MAE by more than 50% on MMSE-HR. While simply subtracting the noise from the signals in the frequency domain often improved the SNR, did not improve the heart rate estimates. Subtracting the noise signal in the time domain performed even worse and had partic- ularly negative impact on the BVP SNR. All results were statistically significant (p- < 0.01) see supplementary material for F-test results.
Heart Rate Estimation.
Table I: Heart rate (HR) and breathing rate (BR) estimation on 3 public datasets. Including the "distraction" regions significantly improves the HR and BR estimation.
Heart Rate
Breathing Rate
AFRL
MMSE-HR
MR-NIRPPNIR)
AFRL
Method
SNR
e
MMA RMSE
/ WMA RMSE SNR
P WMAF
MAR
RMSE
SNR
WMAF
Distraction
2.25 5.68
6.44 0.87
0.21
2.27
4,90
5.00 0.94
0.19
2.34
4.46
2.27
0.85
0.45
2.44
4.23
14,20
0.35
0.28
No Noise
6.86
0.90
2.56
5,23
2.28
0.80
2.49
4.26
14.06
0.27
Freq. Sub
2.92 6.67
3.66
0.82
0.24
3.97
9.93
4.49
0.76
0.57
8.58
17.59
4.56
0.31
5.03
7.45
7.78
0.12
0.31
Wave. Sub
2.92 6.66
3.09
0.82
0.24
6.09
10.84
4.75
0.55
888
17.0
4.69-0.17
0.31
7.21
888
4.12
-0.03
0.37
MAICA
3.91
0.86
RhythmNet
5.49
0.84
PVM
CAN 2.93 6.69
3.36
0.82
0.23
4.06
9.51
0.63
0.77
0.57
7.78
168
-3.24
0.36
4,86
7.32
8.33
0.10
0.27
POS
4.36 9.45
0.73
0.74
0.45
3.90
9.61
2.33
0.78
0.39
Tulyakov
11.37
0.71
Li
19.95
0.38
Tarassenko
3.68
5.52
-6.22
0.29
CHROM 4.07
9.72
0.29
0.72
0.41
3.74
8.11
1.90
0.82
0.37
ICA
0.43
0.66
MAICA (2019) MMcwan al. 2019)
Rythnnee (2019) = (Niu al. 2019.
PVM (2018) (Macwan
2018
CAN (2018)=
Chen
& McDuff 2018) POS (2017)
Wang et al. 2017)
Tulyakov (2016) (Talyakov et al. 2016), (2014)
[L. et
2014
Tarassemka
(2014)
Tarassenko
2014]
CHROM (2013) (De Haan Jeanne 2013), ICA (2010)
Pob et
2010n)
Breathing Rate Estimation. In addition to estimating heart rate, which is based on intensity varia- tions in the skin, method can also be used estimate breathing rate (BR) which is based on mo- tion variations and is more challenging in presence of body motions. Only the AFRL dataset (Estepp) et al. 2014) had gold standard reference breathing signals, therefore we were not able evaluate
7

---

Under review as a conference paper at ICLR 2021
our BR results on the other datasets. Our method achieves reduction in MAE from 3.68 BPM to 2.44 BPM (a 34% error reduction) over the baselines and an increase in SNR of 5.87 dB (Table I Our method also obtains cleaner breathing signals compared to the baseline (Fig. 7
True Benefit of Distraction Regions. Using model without noise estimates from the output of the CAN to the LSTM works well when the signals do not change much over time, and when the noise in the training and test sets is similar, e.g.. training and testing on AFRL (Table I However, including the dis- traction regions yields improvements in both HR and BR estimates when the signal varies over time or there tween training and testing sets. For example, distraction regions improve the performance on MMSE-HR which has sudden pulse variations, uncontrolled motion, and the presence of fa- cial expressions and the more challenging NIR MR-NRIP dataset (Table D. Moreover, including the distraction regions improves the HR and BR estimation accuracy when we train our model only on stationary videos of AFRL (Task 1) and test on videos with large random motions (Task 6) (Table SNR is often higher in the "no noise" condition because it simply produces a smoother signal leading to greater sparsity in the frequency domain. However, the dominant frequency of the signal (used to compute HR and BR) is often erroncous, resulting in higher MAE and RMSE, and lower p. These results show that the distraction signal is useful above and beyond including a temporal component to the model.
RGB Video (AFRL Dataset)
Fames
Current state- of ahe-art (CAN)
Normalzed Amplttie
large domain gap be-
15
20
75
Ours
Time
Narmalized ampiture
25
Black Contact Sensor Measurement
Careera-Besee Estimate
Figure 7: Respiration signals output by a state-of- the-art network and our denoising method. The signals produced by our method are cleaner and match the ground truth better.
Transfer learning. NIR videos of MR-NIRP are more challenging than RGB because the physiological sig- nal is an order of magnitude weaker in the NIR range compared to the vis- ible range. making it very prone to motion artifacts. When trained solely on RGB videos (AFRL dataset) with- out any fine-tuning, our method out- performs all the baselines across all five metrics on the NIR videos from the MR-NIRP dataset. As shown in Table T the MAE drops from 7.78 BPM to 2.34 BPM (70% reduction in error). Other baseline methods require multiple color channels and therefore cannot be applied to NIR videos.
Table 2: Training on AFRL Task and testing on Task 6. The ignored regions help when the training and test set are very different.
Heart Rate
Breathing Rate
Method MAE RMSE SNR
RAEEEEE SNR WMAE
Distraction
5.29
9.33
0.70
0.32
4.28
6.00
No Noise
5.61
9.72
-169
0.32
4.38
6.15
5.96
0.07 0.34
Varying Head Motion. We performed an analysis of the performance motion tasks of AFRL Es- tepp et al. 2014) (see Table Bp. Our method shows improvements over the baseline methods on videos across all head motions. For instance, on videos with angular head rotation of 30 deg/sec (Task 4) the HR MAE was reduced from 2.82 BPM to 1.94 BPM (30% reduction in error) and BR MAE was reduced from 4.85 BPM 2.88 BPM (40% reduction in error).
Inverse Mask Definition. We tested computing the inverse attention mask in two different ways. The first, as a matrix of continuous values in which each element of the inverse mask M, Ma,, was 1- Ar, where A is the attention mask weights normalized from 0 to 1. The second approach was to threshold these values to create binary mask where A 1, if A,,; >T. Where is a threshold from 0 to 1. We found we obtain comparable results with binary (2.25 BPM) or continuous inverse attention masks (2.10 BPM). We also found that the results were not very sensitive the value of T (see supplementary materials).
8

---

Under review as a conference paper at ICLR 2021
Importance of Different Distraction Regions. Certain regions in the video may contain more useful information about the sources of noise than others. For example, regions closer to the face may contain more information about the motion of the participant, while regions farther in the back- ground may contain more information about other sources of noise, such as illumination changes. We compared separately using noise estimates from distraction regions closer to the face (center of the frames) and further from the face (edges of the frames). When motion was small, all regions contributed similarly to denoising (MAE 1.08 BPM with center regions and MAE 1.07 BPM with edges). But when there was large head motion, regions close to the head (center of the frames) helped the most (MAE 6.53 BPM with center regions and MAE 8.74 BPM with edges). See supplementary materials for detailed results.
Effect of Glasses. Interestingly, our method performed best on subjects who wore glasses. The attention masks for subjects with and without glasses were comparably good. How- ever, CAN performed worse sub- jects with glasses and our approach offered large improvement on those videos (MAE [BPM] with glasses: Ours 2.17, CAN 3.33, and with- out glasses: Ours 2.55, CAN 2.57). See supplementary materials for example attention masks and re-
Table 3: Motion increasing from 1 to on AFRL
Heart Rate MAE
Breathing Rate MAR
Meihod
3
Distraction
L06 2. 1.79 2,50 4.78
1.42
1.66
18
2.88
2.87
4.15
Noise
1.14
1.90
3.099.044444
4.52
1.47
1.95
1.68
4.15
Freq. Sub.
1.52 2.51 3.00
5.30
5.30
4,89
5.27
5.09
5.26
Wave, Sub
1.57
2.53 3.03
2.72 5.09
4.31
5,24
5.08
5.19
CAN
1.52 2.51 3.00
2.62
5.34
424551745885044425555
5.0951
POS
1.42 2.84
6.33
10.16
CHROM
1.33
11.86
ICA
2.18 4.74
7.02 13.18
Tar:
2.51
2.53 3.85
sults.
CONCLUSION
We have presented a novel approach for generating noise estimates from inverse attention masks to improve camera-based physiological signal measurements. We hypothesized that the noise affecting regions used by the attention masks to compute the signal of interest would likely be present in other regions in the video which are ignored by the attention masks. Our proposed denoising method out- performed all state-of-the oh-the-art methods in heart rate and breathing rate estimation from videos. The recovered BV signals are also sufficient to recover subtle waveform dynamics present in the ground truth contact signals, including the dicrotic notch and the diastolic peak. Our approach trained on RGB videos showed strong cross-datased and cross-modality generalizability. outperforming the ex- isting methods challenging NIR videos.
REFERENCES
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 6077-6086, 2018
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 2014.
Weixuan Chen and Daniel McDuff. Deepphys: Video-based physiological measurement using con- volutional attention networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 349-365, 2018.
Weixuan Chen, Javier Hernandez, and Rosalind W Picard. Estimating carotid pulse and breathing rate from near-infrared video of the neck. Physiological measurement, 39010), 2018.
Gerard De Haan and Vincent Jeanne. Robust pulse rate from chrominance- rppg. IEEE Trans- actions on Biomedical Engineering, 60(10):878-2888) 2013.
Mohamed Elgendi, Richard Fletcher, Yongbo Liang, Newton Howard, Nigel H Lovell, Derek Ab- bott, Kenneth Lim, and Rabab Ward. The use of photoplethysmography for assessing hyperten- sion. NPJ digital medicine, 2(1):1-11, 2019.

---

Under review as a conference paper at ICLR 2021
Justin R Estepp, Ethan B Blackford, and Christopher M Meier. Recovering pulse rate during motion artifact with a multi-imager array for non-contact imaging photoplethysmography. In 2014 JEEE International Conference Systems, Man, and Cybernetics (SMC), pp.
IEEE, 2014.
Hiroshi Fukui, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi. Attention branch network: Learning of attention mechanism for visual explanation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10705-10714, 2019.
Mare Garbey, Nanfei Sun, Arcangelo Merla, and Ioannis Pavlidis. Contact-frce measurement of cardiac pulse based the analysis of thermal imagery. IEEE transactions Biomedical Engi- neering, 54(8): 101p):118811101 1-266,, 2007.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980 2014.
Eugene Lee, Evan Chen, and Chen-Y Yi Lee. Meta-rppg: Remote heart rate estimation using a trans- ductive meta-leamne. arXiv preprint arXiv:2007.06786 2020.
Xiaobai Li, Jie Chen, Guoying Zhao, and Matti Pietikainen. Remote heart rate measurement from face videos under realistic situations. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2014.
Siqi Liu, Xiangyuan Lan, and PongChi Yuen. Temporal similarity analysis of remote photoplethys- mography for fast 3d mask face presentation attack detection. In The IEEE Winter Conference Applications of Computer Vision, pp. 2608-2616, 2020a.
Xin Liu, Josh Fromm, Shwetak Patel, and Daniel McDuff. Multi-task temporal shift attention net- works for on-device contactless vitals measurement. arXiv preprint arXiv:2006.0399) 2020b.
Richard Macwan, Serge Bobbia, Yannick Benezeth, Julien Dubois, and Alamin Mansouri. Peri- odic variance maximization using generalized eigenvalue decomposition applied to remote pho- toplethysmography estimation. In Proceedings of the IEEE Conference on Computer Vision and Pastern Recognition Workshops, pp. 1332-1340, 2018.
Richard Macwan, Yannick Benezeth, and Alamin Mansouri. Heart rate estimation using remote pho- toplethysmography with multi-objective optimization. Biomedical Signal Processing and Con- Irol, 49:24-33, 2019.
Daniel McDuff. Deep super resolution for recovering physiological information from videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1367-1374, 2018
Daniel McDuff, Javier Hernandez, Sarah Gontarek, and Rosalind W Picard. Cogcam: Contact-free measurement of cognitive stress during computer tasks with a digital camera. In Proceedings of the 2016 CHI Conference Human Factors in Computing Systems, pp. 4000-4004. ACM, 2016.
Daniel J McDuff, Ethan B Blackford, Justin R Estepp. and Izumi Nishidate. A fast non-contact imaging photoplethysmography method using tissue-like model. In Optical Diagnostics and Sensing XVIII: Toward Point-of-Care Diagnostics, volume 10501, pp. 105010Q. International Society for Optics and Photonics, 2018.
Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In Advances in neural information processing systems, pp. 2204-2212, 2014.
Xuesong Niu, Hu Han, Shiguang Shan, and Xilin Chen. Synrhythm: Learning a deep heart rate estimator from general to specific. In 2018 24th International Conference on Pattern Recognition (ICPR), pp. 3580-3585. IEEE, 2018a.
Xuesong Niu, Hu Han, Shiguang Shan. and Xilin Chen. Vipl-hr: A multi-modal database for pulse estimation from less-constrained face video. Asian Conference on Computer Vision, pp. 562- 576. Springer, 2018b.
10

---

Under review as a conference paper at ICLR 2021
Xuesong Niu, Shiguang Shan, Hu Han, and Xilin Chen. Rhythmnet: End-to-end heart rate estima- tion from face via spatial-tempore representation. IEEE Transactions on Image Processing, 29: 2409-2423, 2019.
Ewa Nowara and Daniel McDuff. Combating the impact of video compression on non-contact vital sign measurement using supervised learning. In Proceedings of the IEEE International Conference on Computer Vision Workshops, 2019.
Ewa Magdalena Nowara, Ashutosh Sabharwal, and Ashok Veeraraghavan. Ppgsecure: Biometric presentation attack detection using photoplethy.moorans. In 2017 12th IEEE International Con- ference Automatic Face & Gesture Recognition (FG 2017), pp. 56-62. IEEE, 2017.
Ewa Magdalena Nowara, Tim K Marks, Hassan Mansour, and Ashok Veeraraghavan. Sparseppg: towards driver monitoring using camera-based vital signs estimation in near-infrared. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp- 1353-135309. IEEE, 2018.
Aude Oliva, Antonio Torralba, Monica Castelhano, and John M Henderson. Top-down control of visual attention in object detection. In Proceedings 2003 International Conference Image Processing (Cat. No. 03CH37429), volume I, pp. I-253. JEEE, 2003.
I Pavlidis, M Dcosta, S Taamneh, M Manser, T Ferris, R Wunderlich, E Akleman, and P Tiimmyytzis.. Dissecting driver behaviors under cognitive, emotional, sensorimotor, and mixed stressors. Sci- entific reports, 6:25651, 2016.
Ming-Zhee Zher Poh, Daniel McDuff, and Rosalind W Picard. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Optics express, 18(10):10622- 10774, 2010a.
Ming Ming-Zher Poh, Daniel J McDuff, and Rosalind W Picard. Advancements in noncontact, multi- parameter physiological measurements using webcam. IEEE transactions on biomedical engi- neering, 58(1):7-11, 2010b.
Valentina Puntmann, Ludovica Carerj, Imke Wieters, Masia Fahim, Christophe Arendt, Jedrzej Hoffmann, Anastasia Shchendrygina, Felicitas Escher, Mariuca Vasa-Nicotera, Andreas M Zei- her, et al. Outcomes of cardiovascular magnetic resonance imaging in patients recently recovered from coronavirus disease 2019 (covid-19). JAMA cardiology. 2020.
Dangdang Shao, Yuting Yang, Chenbin Liu, Francis Tsow, Hui Yu, and Nongjian Tao. Noncontact monitoring breathing pattern, exhalation flow rate and pulse transit time. IEEE Transactions Biomedical Engineering,
2767, 2014.
Shikhar Sharma, Ryan Kiros, and Ruslan Salakhutdinov. Action recognition using visual attention. arXiv preprint 1XX1::1511.04 19, 2015.
Radim Spetlik, Vojtech Franc, and Jirí Matas. Visual heart rate estimation with convolutional neural network. In Proceedings of the British Machine Vision Conference, Newcastle, UK, pp. 3-6, 2018.
Chihiro Takano and Yuji Ohta. Heart rate measurement based on time-lapse image. Medical engineering & physics, 29888333-8 2007.
L Tarassenko, M Villarrel, A Guazzi, J Jorge. DA Clifton, and C Pugh. Non-contact video-based vital sign monitoring using ambient light and auto-regressive models. Physiological measurement, 35(5):807, 2014.
Mika P Tarvainen, Perttu O Ranta-Aho, and Pasi Karjalainen. An advanced detrending method with application hrv analysis. IEEE Transactions on Biomedical Engineering, 4722):172-175. 2002.
An Tran and Loong-Fah Cheong. Two-stream flow-guided convolutional attention networks for action recognition. Proceedings of the IEEE International Conference on Computer Vision, pp. 3110-3119 2017.
11

---

Under review as a conference paper at ICLR 2021
Sergey Tulyakov, Xavier Alameda-Pineda, Elisa Ricci, Lijun Yin, Jeffrey F Cohn, and Nicu Sebe. Self-adaptive matrix completion for heart rate estimation from face videos under realistic condi- tions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 2396-2404, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is you need. Advances in neural information processing systems, pp. 5998-6008 2017.
Wim Verkruysse, Lars O Svaasand, and Stuart Nelson. Remote plethysmographic imaging using ambient light. Optics express, 16(26):21 112221144-21444 2008
Wenjin Wang, Albertus C den Brinker, Sander Stuijk, and Gerard de Haan. Algorithmic principles of remote ppg. IEEE Transactions Biomedical Engineering, 64(7):1479-144 2017.
Zhe Xu, Lei Shi, Yijin Wang, Jiyuan Zhang, Lei Huang, Chao Zhang, Shuhong Liu, Peng Zhao, Hongxia Liu, Li Zhu, et al. Pathological findings of covid-19 associated with acute respiratory distress syndrome. The Lancet respiratory medicine, 8(4):420-422 2020.
Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In Proceedings of the IEEE conference computer vision and pattern recog- nition, pp. 4651-4659, 2016.
Zitong Yu, Wei Peng, Xiaobai Li, Xiaopeng Hong, and Guoying Zhao. Remote heart rate mea- surement from highly compressed facial videos: end-to-end deep learning solution with video enhancement. In Proceedings of the IEEE International Conference on Computer Vision, pp. 151-160, 2019.
Qi Zhan, Wenjin Wang, and Gerard de Haan. Analysis of enn-based remote-ppg to understand limitations and sensitivities. arXiv preprint arXiv: 1et::9911.0206, 2019.
Zheng Zhang, Jeff M Girard, Yue Wu, Xing Zhang, Peng Liu, Umur Ciftei, Shaun Canavan, Michael Reale, Andy Horowitz, Huiyuan Yang, et al. Multimodal spontaneous emotion corpus for hu- behavior analysis. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3438-3446, 2016.
Ying- Ying Zheng, Yi-Tong Ma, Jin- Ying Zhang, and Xiang Xie. Covid-19 19 and the cardiovascular system. Nature Reviews Cardiology, 17(5):259-260 2020.
12


Reviews:
[{"review_id": "eLNamEigMSh", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. It is interesting to model the noises mixed with the very weak physiological signal; however, the assumption used in this paper, i.e., most regions outside of the attention masks do not contain the signals of interest and consequently contain noise does not really hold, because, besides the weak physiological signal, the dominant signal inside the face video is the face content, i.e., the texture of the face describing the appearance of a person; these background signal dominant the video but cannot be treated as noises.\n\n2. The novelty of the proposed method is quite limited. The general idea is to use an inverse mask, i.e., {1- attention mask}, to determine the regions less correlated with physiological signal, within which the average intensity is computed and treated as noise. However, as we know that for Gaussian-like noises, averaging the local pixel is to remove the noise instead of obtaining the noise. Therefore, the claimed method to compute noise is not well motivated and explained.\n\n3. The experiments are performed on three datasets; however, the authors seem to not follow the widely used protocols but defined a new protocol instead. As a result, even though there a number of SOTA methods in this topic as listed in the references, they were not used to compare with the proposed approach. So it is not clear how the proposed approach can advance the methodology. In addition, while several large physiological signal datasets are available, e.g., with 100-500 subjects, the authors did not use these datasets for evaluations.\n\n4. It is not clear how the authors can measure an output wavelet match the ground-truth BVP signal? particularly when there is misalignment between the video and the finger pulse oximeter wave.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MEcjIJKSjwF", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new method for the camera based physiological measurement task. The key idea of this work is to use attention mechanism to learn discriminative features from regions of interest, and use the inverse attention mask to select contextual information to learn noise representation for refinement. Experiments on three datasets show state-of-the-art performance.\n\nI have several main concerns.\n\n1. The main idea of using the reverse attention mask to learn noise information is not significant. In fact, using reverse attention to focus on other regions has been studied in other areas (e.g., for saliency detection). The paper applies the idea to a specific task, i.e., the camera based physiological measurement task.\n\n2. The motivation is not well justified. It is not explained the advantages of using reverse attention mask to learn noise representation. Taking Figure 1 for example, it seems that the attention mask focuses on some key facial regions. While these regions of interest also contain noisy informaiton (e.g., motion and lighting change), the reverse attended regions (the textured background) may not necessarily contain. Why not directly learn an attention map for noise estimatation, or why not directly use the original attention map to learn noise representation?\n\n3. The discussion of how existing methods deal with large motion and noise is not given in the related work section. It is then hard to evaluate the significance of proposed method.\n\n4. The computation of noise (Eq.1) is a multiplication of the reverse attention mask and the original input. However, I do not understand how this operation can be interpreted as noise estimation, as it seems to be a region selection process. Given that, the denoising model seems to be a refinemet model that considers reverse attended regions and the output of the first model. It is hard for me to understand the second model as a denoising model. \n\nMinor issues.\n\n1. The description of Fig.1 can be improved with more details.\n\n2. It is better to provide more details of how errors are corrected in the MMSE-HR dataset (Last sentence of the first paragraph in page 6).\n\n3. Section 5 talks about the training details. It would be better to directly use \"Training Details\" instead of \"Experiments\".\n\n4. In the first paragraph of section 6, four variants are given. However, it is not easy to understand the last two, i.e., how the noise subtraction is performed in frequency or time domain. \n\n5. Tables 3 -> Table 3 (page 8).\n\n6. The abstract mentions two datasets but there are actually three datasets in the experiments. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8-nlLIpe17I", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The article proposes a model for estimating physiological signals from videos. The novelty of the proposed approach is to use the low attention regions to estimate a model of the signal noise. Using this estimate can improve the performance of the denoising component. \n\nThe model is well described and sensible and the experimental section demonstrates state-of-the-art results on what is a very relevant task.\n\nOverall, the article describes an interesting approach and contains valuable insights into the limitations of attention-based approaches in noisy signals.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a procedure that uses low attention areas to de-noise temporal prediction. The paper appears to focus on 'temporal noise' as opposed to constant noise present in the video (it may handle shifting shadows, but not background noise). 

The idea is certainly interesting, however, the experimental protocol suffers from the issues pointed out by reviewer 3:
- maintaining the same protocol as prior methods to ensure a direct comparison of the results against reported scores by the sota
- in the context of attention, alignment (or lack or it) is extremely important; assuming perfect alignment is not very realistic
(if the alignment is perfect, one might try a simple method such as taking all readings at a point over time and considering the mode, then correcting any outliers in the off-attention areas)

These specific issues were not fully addressed during the review period.

The questions raised by reviewer 2 were addressed to a satisfactory degree in the rebuttal.
Decision: Reject