# Spatiotemporal Sensitive Network for Non-Contact Heart Rate Prediction from Facial Videos
## 3 Proposed Method
### Spatiotemporal Sensitive (SS) Module
Thus, the attention computation in the new windows crosses the boundaries of the windows in layer \(I\), providing connections among them. Finally, the output of the \(l\)-th SS module is calculated as follows:  
Figure 3: The SS (spatiotemporal sensitive) module with an SS block; the left side represents the SS module, and the right side represents the SS block. \[ \begin{array}{l}\overset{\wedge}{z}l=W-MSA(SS(LN(z_{l-1})))+z_{l-1}\\ \overset{\sim}{z}l=MLP(LN(\overset{\wedge}{z}l))+\overset{\sim}{z}l\\ z\overset{l}=SW-MSA(SS(LN(\overset{\wedge}{z}l)))+\overset{\sim}{z}l\\ z_{l}=MLP\big{(}LN(\overset{\wedge}{z}l)\big{)}+\overset{\sim}{z}l\end{array}  \tag{3}\]  
where \(\overset{\wedge}{z}l,\overset{\sim}{z}l\) and \(\overset{\sim}{z}l\), respectively, represent the outputs of W-MSA, MLP, and SW-MSA. \(z_{l}\) denotes the output of the \(l\)-th SS module. In the Swin transformer [28], whether there is W-MSA or SW-MSA, the window size is always fixed, limiting the window to only contain feature maps within a fixed temporal and spatial range. As observed in [29], the size of the input feature map affects the network's final output because a feature map that is too large contains excessive noise, while a feature map that is too small lacks sufficient useful information. To address this limitation, we propose adding an SS block before the MSA module. The design of the SS block allows the MSA to interact with multiple spatiotemporal dimensions in a fixed-size window calculation, enhancing its sensitivity to physiological information in different spatiotemporal dimensions. The structure of the SS block is shown on the right side of Figure 3. We first shuffle the channels of the spatiotemporal feature map evenly [30], ensuring that after the subsequent split group operation, the feature map retains the information exchange between different groups. Then, we group the spatiotemporal feature map along the channel direction. Within each group, we use dilated convolution [31] to extract features from different spatiotemporal coordinate ranges. Finally, we concatenate the grouped spatiotemporal feature maps back to their original shape, enabling the spatiotemporal feature map to encompass features from multiple spatiotemporal coordinate scales. Additionally, we introduce coordinate attention [32], which applies attention along each of the two coordinate directions independently. This allows it to capture long-range dependencies along one coordinate direction while preserving precise positional information along the other, thereby helping the network learn spatiotemporal information across different spatiotemporal dimensions in the feature map. Specifically, \(F\) represents the input to the SS block. Initially, \(F\) undergoes a channel shuffle operation, where the channel order is uniformly disrupted and equally divided into \(S_{1}\), \(S_{2}\), and \(S_{3}\). Then, \(S_{1}\), \(S_{2}\) and \(S_{3}\) are passed through three parallel branches of dilated convolution, each with a dilation rate of \(r=1,r=3,\text{ and }r=5\). The outputs of these branches, \(x_{1},\ x_{2}\) and \(x_{3}\), are concatenated and the channel number is restored to c, resulting in \(x\). The outputs of the three branches are expressed as  
\[ \begin{array}{l}S_{1},\ S_{2},\ S_{3}=Split(Shuffle(F))\\ x_{1}=DilatedConv_{r=1}(S_{1})\\ x_{2}=DilatedConv_{r=3}(S_{2})\\ x_{3}=DilatedConv_{r=5}(S_{3})\\ x=Concat(x_{1},x_{2},x_{3})\end{array}  \tag{4}\]  
Given the input feature \(x\in R^{\text{C}\times H\times W}\), we use two spatial extents of pooling kernels \((H,1)\) or \((1,W)\) to encode each channel along the horizontal coordinate and the vertical coordinate, respectively.