# Spatiotemporal Sensitive Network for Non-Contact Heart Rate Prediction from Facial Videos
## 3 Proposed Method
### Spatiotemporal Sensitive (SS) Module  
The skin color of the face changes slightly over a continuous period of time as the pulse contracts [13; 27], which indicates that there is a time dependency in predicting skin color changes between frames. To obtain a more robust heart rate measurement method, we focus on the contextual relationship of skin color changes in continuous time based on the principle that the signal characteristics of rPPG are obtained through skin color changes in continuous time. We design a special attention module in the rPPG field based on a transformer, named the spatiotemporal sensitive (SS) module, which can obtain rPPG-sensitive information in both time and space dimensions, and the slight changes in skin color can be obtained sensitively. On this basis, we introduced a multi-channel dilated convolution design in the field of attention mechanism. Through receptive fields of different sizes, we can more comprehensively capture the changes in global and local features in the spatiotemporal dimensions, and further improve the rPPG's ability to capture signals of different degrees of changes. The proposed SS module consists of a convolutional detector and an attention screening mechanism. The convolutional detector employs dilated convolutions to incorporate signals from different receptive fields into the signal pool. Subsequently, the attention screening mechanism, utilizing a coordinate attention mechanism, filters these signals, enabling the selection of physiologically sensitive areas in the spatiotemporal feature maps. As shown in Figure 1, RGB and NIR spatiotemporal maps are input into SS modules with identical structures but different parameters. The specific structure of the spatiotemporal  
Figure 2: The process of converting the original video sequence into spatiotemporal feature maps. poral sensitive (SS) module is depicted in Figure 3, with each SS module containing 2 SS blocks. Given features \(F\) with the size of \(H\times W\times C\), the shifted window mechanism first reshapes the input to \(\frac{HW}{M^{2}}\times M^{2}\times C\) features via partitioning the input into non-overlapping \(M\times M\) local windows, where \(\frac{HW}{M^{2}}\) is the total number of windows. Next, it performs standard self-attention separately for each window. For a local window feature \(X\in\mathbb{R}^{M^{2}\times C}\), three learnable weight matrices, \(\mathbb{W}^{Q}\in\mathbb{R}^{C\times C}\), \(\mathbb{W}^{K}\in\mathbb{R}^{C\times C}\) and \(\mathbb{W}^{V}\in\mathbb{R}^{C\times C}\), that are shared across different windows are employed to project it into the query Q, key K and value V by:  
\[\{\mathbb{Q},\mathbb{K},\mathbb{V}\}=\left\{X\mathbb{W}^{Q},X\mathbb{W}^{K},X \mathbb{W}^{V}\right\} \tag{1}\]  
Then, the attention function computes the dot-product of the query with all keys, which is then normalized with the softmax operator to yield attention scores. The attention mechanism is defined as:  
\[\text{Attention}(\mathbb{Q},\mathbb{K},\mathbb{V})=\text{softmax}\left(\frac{ \mathbb{Q}\mathbb{K}^{T}}{\sqrt{d_{k}}}+\mathbb{B}\right)\mathbb{V} \tag{2}\]  
where \(d_{k}\) is the dimension of keys and B is the learnable relative positional encoding. We extend the self-attention into window multi-head self-attention (W-MSA) or shifted window multi-head self-attention (SW-MSA) to enable the attention mechanism to consider various attention distributions and make the model capture information from different perspectives. The spatiotemporal sensitive (SS) block is always performed after both layer normalization (LN) and MSA. Next, LN and multi-layer perceptron (MLP) layers are deployed to refine the feature tokens yielded by MSA. As can be seen in the green flowchart on the left side of Figure 3, in layer \(I\), a window multi-head self-attention (W-MSA) layer is employed, and the attention is computed within each window. In the next layer, a shifted window multi-head self-attention (SW-MSA) layer is employed, which results in new window.