# Real-Time Webcam Heart-Rate and Variability Estimation with Clean Ground Truth for Evaluation +
## 2 Related Work  
Signal processing based rPPG methods.Since the early work of [PERSON] et al. [3], who showed that heart rate could be measured from consumer grade camera recordings in ambient light, a large body of research has been conducted on the topic. Extensive reviews of these work can be found in [2; 13; 15]. Most published rPPG methods work either by applying skin detection on a certain area in each frame or by selecting one or multiple regions of interest and track their averages over time to generate colour signals. A general division can be made into methods that use blind source separation (ICA, PCA) [16; 17; 18] vs. those that use a 'fixed' extraction scheme for obtaining the blood volume pulse (BVP) signal [19; 20; 21; 22; 23]. The blind source separation methods require an additional selection step to extract the most informative BVP component signal. To avoid this, we make use of a 'fixed' extraction scheme in our method. Among the 'fixed' methods, multiple stand out and serve as inspiration and foundation for this work. [PERSON] et al. [21] presented the first face modelling based signal extraction method and utilized detrending [24] based filtering to estimate BVP and heart rate. The CHROM [19] method uses a ratio of chrominance signals which are obtained from RGB channels followed by a skin-tone standardization step. [PERSON] et al. [20] proposed an extra illumination rectification step using the colour of the background to counter illumination variations. The SAMC [22] method proposes an approach for BVP extraction in which regions of interest are dynamically chosen using self adaptive matrix completion. The Plane-orthogonal to skin (POS) [23] method improves on CHROM. It works by projecting RGB signals on a plane orthogonal to a normalized skin tone in normalized RGB space, and combines the resulting signals into a single signal containing the photopleytismographic information. We take inspiration from [PERSON] et al. [21] and further build upon POS [23]. We introduce additional signal refinement steps for accurate peak detection to further improve HR and HRV analysis. Deep learning based rPPG methods.Most recent works have applied deep learning to extract either heart rate or the blood volume pulse directly from camera images. They rely on the ability of deep networks to learn which areas in the image correspond to heart rate. This way, no prior domain knowledge is needed and the system learns the underlying rPPG mechanism from scratch. DeepPhys [5] is the first such end-to-end method to extract heart and breathing rate from videos. HR-CNN [6] uses two successive convolutional neural networks (CNNs) [25] to first extract a BVP from a sequence of images and then estimate the heart rate from it. RhythmNet [26] uses a CNN and gated recurrent units to form a spatiotemporal representation for HR estimation. The recent work of AutoHR [27] employs neural architecture search to discover temporal difference convolution as a strong backbone to capture the rPPG signal from frame sequences. These methods have shown state-of-the-art performance on multiple public and private datasets. Our presented algorithm is unsupervised and makes use of an active appearance model [28] to select regions of interest to extract a heart rate signal from. Due to this, no rPPG specific model training is required while prior domain knowledge is more heavily relied upon. Heart Rate Variability from rPPG.Some past methods have also attempted extracting heart rate variability from videos [18; 29; 30]. A good overview of this is provided by [PERSON] et al. [31]. Because HRV is calculated based on variations in inter-beat intervals, it is crucial that single beats are detected and localized with a high degree of accuracy. Methods that otherwise show good performance in extracting HR can be unsuitable for HRV analysis since they may not provide beatlocations. [PERSON] et al. [31] evaluate their baseline rPPG method for HRV estimation. Their method is based on bandpass filtering the green channel from regions of interest.