# Spatiotemporal Sensitive Network for Non-Contact Heart Rate Prediction from Facial Videos
## 3 Proposed Method
### Multi-Level Spatiotemporal Context Fusion (MLSC) Module  
To solve the problem of the poor effect of non-contact measurement in complex environments, we complement RGB and NIR modal information to improve the robustness of measuring heart rate. Most feature fusion techniques are limited to integrating complementary information from a local perspective and fail to accomplish comprehensive global information aggregation. Such fusion methods lose the contextual relationship of information across continuous time segments and adjacent spatial ranges. Based on [33], we developed a spatiotemporal context fusion (SC) module, which can effectively exploit and aggregate the global spatiotemporal context within and between modalities, thereby enhancing the complementary fusion of visible light and infrared data. Furthermore, inspired by MTAN [34], we proposed a multi-level fusion framework, named the multi-level spatiotemporal context fusion (MLSC) module, which consists of three SC modules to perform signal fusion at multiple semantic levels. These fused features with different semantic levels are used in subsequent specific task branches, and each branch assigns the best weight to them to achieve the best effect for a specific task. The spatiotemporal context fusion (SC) module we designed is illustrated in Figure 4. The key difference from the Swin transformer lies in its use of multi-head cross-attention (MCA) instead of MSA (multi-head self-attention), enabling cross-modal global context interaction. Stage 1 of the SC module focuses on intra-modal fusion to integrate global interactions in the same modality and stage 2 focuses on inter-modal fusion to further integrate the global interactions between different modalities. The structure of stage 1 of the SC module differs from that of the SS module by the presence or absence of the SS block, so we will not repeat the explanation of the first stage of the SC module. Assuming the outputs of the first stage of the SC module are \(X1\) and \(X2\), stage 2 of the SC module is defined as follows:As shown in Equation (10), for the Q\({}_{1}\) from the RGB modality, it integrates cross-modal information by performing attention weighting with K\({}_{2}\) and V\({}_{2}\) from the NIR modality, while preserving its modality information in the RGB modality through the residual connections. To balance computational resources and fusion effectiveness, our model deploys three SC modules. Each fusion layer consists of cascaded intra-modal fusion units and inter-modal fusion units, alternating to integrate global intra-modal and inter-modal interactions. After each spatiotemporal context fusion layer, we obtain outputs \(Z_{1}^{i}\) and \(Z_{2}^{i}\) from the two modalities, where \(0<i<4\) denotes the output of the \(i\)-th SC module. We concatenate \(Z_{1}^{i}\) and \(Z_{2}^{i}\) along the channel dimension and then aggregate local information from different modalities using a convolutional layer with spatially invariant filters. This enhances our model's translational equivariance and can be represented as follows:  
\[ \begin{array}{l}Z_{level1}=H_{conv}\big{(}\mathcal{C}oncat\big{(}Z_{1}^{1},Z_ {1}^{1}\big{)}\big{)}\\ Z_{level2}=H_{conv}\big{(}\mathcal{C}oncat\big{(}Z_{1}^{1},Z_{2}^{1}\big{)} \big{)}\\ Z_{level3}=H_{conv}\big{(}\mathcal{C}oncat\big{(}Z_{1}^{3},Z_{2}^{3}\big{)} \big{)}\end{array}  \tag{11}\]  
\(H_{Conv}(\cdot)\) represents a convolutional layer, and \(Concat(\cdot,\cdot)\) denotes concatenation along the channel dimension. \(Z_{leveli}\) refers to the deep features generated by the \(i\)-th SC module. These serve as inputs to the Refine module [35]. The structure of the Refine module is illustrated in Figure 5. We use two independent branches to predict rPPG and HR respectively. Each task branch includes three Refine modules. The Refine module refines the fusion features at each level to extract task-oriented features to improve estimation performance.