# Two-Path Spatial-Temporal Feature Fusion and View Embedding for Gait Recognition
## 3 Proposed Method
### Two-Path Spatial-Temporal Feature Fusion Module
#### 3.2.3 Multi-Scale Temporal Feature Extraction (MSTFE)  
MSTFE is devised to integrate long-term and short-term features in the temporal dimension, facilitating effective information exchange between different scales. As depicted in Figure 5, firstly, the HP operation is implemented on \(F_{MSFE}\) to obtain fine-grained information and then, on the one hand, the long-term features are extracted to represent the motion characteristics of all frames, which reveals the global motion periodicity of different body parts. On the other hand, two sequential 1D convolutions with a kernel size of 3 are used to extract short-term temporal contextual features that are favorable for modeling micromotion patterns. Finally, an adaptive temporal feature fusion approach is employed to acquire the temporal importance weights for each temporal scale at both long-term and short-term scales to adaptively highlight or suppress features to obtain the most discriminative motion characteristics.  
The input of MSTFE is denoted as \(F_{MSFE}\), and the HP operation is used to partition the human body into \(K\) segments, resulting in the temporal feature representation \(F_{Temp}\in R^{C\times T\times K}\), \(F_{Temp}^{i}\) denotes the \(i\)-th sample of each frame. Initially, a multilayer perceptron (MLP) followed by a Sigmoid function is applied to each frame to obtain the long-term temporal features. The MLP could be considered as a temporal attention mechanism, learning to assess the importance of different frames based on their relevance to the overall motion characteristics. Subsequently, the long-term temporal feature \(F_{L}\) is obtained by performing a weighted summation of all frames. The process can be expressed below:  
\[F_{L}=\frac{\sum_{i=1}^{l}Sigmoid\Big{(}MLP\Big{(}F_{Temp}^{i}\Big{)}\odot F_ {Temp}^{i}\Big{)}}{\sum_{i=1}^{l}Sigmoid\Big{(}MLP\Big{(}F_{Temp}^{i}\Big{)} \Big{)}}. \tag{7}\]  
where \(\odot\) denotes dot product and \(F_{L}\) describes the global motion cues.  
To obtain the short-term temporal features, two sequential 1D convolutions with a kernel size of 3 are applied to the temporal feature representation \(F_{Temp}\) obtained from the HP operation. After each 1D convolution, the resulting features are summed to capture the short-term temporal feature \(F_{S}\), which is formulated as:  
\[F_{s}=Conv1d\big{(}F_{Temp}\big{)}\oplus Conv1d\big{(}Conv1d\big{(}F_{Temp} \big{)}\big{)} \tag{8}\]  
To capture the varying motion patterns of different temporal scales and their varying importance in gait recognition, an adaptive temporal feature fusion approach is introduced. This approach aims to determine temporal importance weights that adaptively highlight or suppress features to obtain the most discriminative motion characteristics, which is realized through two fully connected layers followed by a Sigmoid function [35,36]. These layers are specifically designed to learn the temporal importance weights for both long-term and short-term temporal scales. The process can be expressed as follows:  
\[W_{T}=Sigmoid(FC(FC(cat(F_{s},F_{L})))) \tag{9}\]  
The fused temporal features can be realized by:  
\[F_{A}^{n}=F_{L}^{n}\times W_{T,1}^{n}+F_{S}^{n}\times W_{T,2}^{n} \tag{10}\]  
where \(W_{T}^{n}\) denotes the importance weight of the \(n\)-th frame of a sample, and the weights of the long-term and short-term temporal feature are denoted as \(W_{T,1}^{n}\), \(W_{T,2}^{n}\).  
Based on the adaptive temporal feature fusion, we obtain sequence-level representations in a weighted summation way as follows:  
\[F_{SL-term}=\frac{\sum_{n=1}^{N}F_{A}^{n}}{\sum_{n=1}^{N}\sum_{i=1}^{i}Z_{N}^ {n}}. \tag{11}\]  
Figure 5: The detailed structure of multi-scale temporal feature extraction (MSTFE).  
where \(F_{SL-term}\in R^{\mathbb{C}_{2}\times K}\), and \(C_{2}\) means the number of channels.