# Spatiotemporal Sensitive Network for Non-Contact Heart Rate Prediction from Facial Videos
## 1 Introduction  
Heart rate (HR) is a crucial parameter in assessing cardiac physiological activities. Current measurement technologies primarily include electrocardiography (ECG) and photoplethysmography (PPG). However, these methods require contact with the subject's skin, which can cause discomfort and are unsuitable for burn patients or newborns. In recent years, remote photoplethysmography based on facial video has gained significant attention. This method utilizes ordinary cameras to record facial skin color changes caused by cardiac pulsation, enabling the non-contact measurement of heart rate. Existing non-contact signal measurement methods are divided into blind source separation (BSS) methods and optical modeling methods. BSS aims to extract target signals from mixed observations, and common BSS methods include independent component analysis (ICA) [1] and principal components analysis (PCA) [2, 3, 4]. Optical modelingmethods, such as CHROM [5], POS [6], and those in [7], aim to design skin reflection models to reduce the effects of illumination and motion fluctuations on irrelevant noise in rPPG signals. These methods reported good results under constrained situations, but may not perform well in the presence of large head motions and lighting variations. In recent years, deep learning has demonstrated satisfactory performance across various fields, with research on deep learning-based non-contact heart rate measurement becoming a focal point in this domain. Approaches such as rPPG-MAE [8], PhysFormer++ [9], and MaskFusionNet [10] improve the measurement effect by building a suitable neural network, extracting features and removing noise, reducing the impact of motion and lighting changes. Furthermore, [PERSON] et al. [11] have proposed using near-infrared (NIR) videos as input data to neural networks for heart rate prediction, achieving better results under dim lighting conditions. However, most of the existing methods concentrate on single-modal data, which restricts the adaptability and efficacy of the methods by limiting the effective usage of complementary information across distinct modalities. The above-mentioned fact inspires us to advocate for the combination of RGB and NIR modalities for heart rate measurement in complex situations. Multimodal fusion combines information taken from many modalities into a stable multimodal representation in which the information from each modality complements one another. This approach facilitates HR estimation under less-constrained conditions, such as varying lighting, random motion, and skin color differences. Leveraging the complementary advantages of these two modalities, we have developed a spatiotemporal sensitive network for non-contact physiological signal estimation. Initially, we construct spatiotemporal feature maps to acquire a more lightweight and information-rich representation of RGB and NIR video data. These spatiotemporal feature maps are then fed into the SS-Net. Specifically, we introduce a spatiotemporal sensitive (SS) module to extract features from the spatiotemporal feature maps of each single modality. Subsequently, we design a multi-level spatiotemporal context fusion (MLSC) module to integrate features from both modalities across spatiotemporal and hierarchical dimensions, thereby simultaneously outputting multiple physiological signals. We summarize the main contributions of this paper as follows:  
* We constructed a spatiotemporal sensitive network to enhance heart rate prediction under less-constrained environments, which leverages multimodal data and extracts information across different spatiotemporal coordinates in the spatiotemporal feature map. * We designed a spatiotemporal sensitive (SS) module based on a convolution detector and attention screening mechanism to address the practical problem of information scarcity in time and space dimensions.