# Two-Path Spatial-Temporal Feature Fusion and View Embedding for Gait Recognition
## 3 Proposed Method
### Two-Path Spatial-Temporal Feature Fusion Module
#### 3.2.2 Frame-Level Spatial Feature Extraction (FLSFE)  
To further enhance the granularity of spatial-temporal learning and capture contextual information, frame-level spatial feature extraction (FLSFE) is constructed to extract both global and local spatial features for each frame. The structure of FLSFE is shown in Figure 3. The input of FLSFE is denoted as \(F_{MSFE}\in R^{C\times T\times H/2\times w/2}\), and FLSFE is composed of three consecutive R-convs (detailed structure could be found in Figure 4), each of which is designed to extract both the whole-body and part-informed spatial features. Then, set pooling operation (SP) is applied to each R-conv block to extract set-level features, and after point-by-point summation, column vector spatial features \(F_{space}\in R^{C_{1}\times K}\) are obtained by the horizontal pooling operation (HP), where \(C_{1}\) means the number of channels, and \(K\) represents the number of strips cut in HP.  
Figure 4 illustrates the structure of the R-conv, which is a novel residual convolutional block that combines regular convolution with FConv in parallel to extract both global and  
Figure 4: The detailed structure of R-conv.  
Figure 3: The detailed structure of frame-level spatial feature extraction (FLSFE).  
Figure 2: The detailed structure of multi-scale feature extraction (MSFE).  
local features from gait sequences. The R-conv is designed to leverage the benefits of both global and local features by adding the output of FConv with the result of regular convolution. Assuming that the input is denoted as \(X_{global}\in R^{c_{1}\times t\times h\times w}\), where \(c_{1}\) represents the number of channels, \(t\) means the length of feature maps and \((h,w)\) indicate the dimensions of each image frame. Initially, the global feature map is horizontally partitioned into \(n\) parts to derive local feature maps, represented as \(X_{local}=\big{\{}X_{local}^{i}|i=1,\cdots,n\big{\}}\), where \(n\) denotes the total number of partitions and \(X_{local}^{i}\in R^{c_{1}\times t\times\frac{n}{h}\times w}\) represents the \(i\)-th local gait segment. Subsequently, 2D CNN is used to extract global and local gait information independently. Finally, the combined features \(F_{R-conv}\), which includes global and local information, are fused through element-wise summation, formulated as:  
\[F_{R-conv}=F_{global}+F_{local}\in R^{c_{2}\times t\times h\times w} \tag{2}\]  
\(F_{global}\) and \(F_{local}\) can be expressed as:  
\[F_{global}=F_{3\times 3}\Big{(}X_{global}\Big{)}\in R^{c_{2}\times t\times h \times w} \tag{3}\]  
\[F_{local}=cat\left\{\begin{matrix}F_{3\times 3}\big{(}X_{local}^{1}\big{)} \\ F_{3\times 3}\big{(}X_{local}^{2}\big{)}\\ \vdots\\ F_{3\times 3}\big{(}X_{local}^{n}\big{)}\end{matrix}\right\}\in R^{c_{2}\times t \times h\times w} \tag{4}\]  
where \(F_{3\times 3}\) represents the 2D convolution operation with the kernel size of (3,3), and \(cat\) denotes the concatenation operation on the horizontal dimension.  
In FLSFE, except for three consecutive R-convs, SP is used to aggregate high-level feature maps from different gait timepoints that are robust to the appearance and observation perspectives into set-level feature maps. The set-level spatial representation is generated as \(F_{SP}\in R^{C\times h\times w}\) and the process can be formulated as Equation (5):  
\[F_{SP}=SP(F_{R-conv}) \tag{5}\]  
Then, set-level features are summed up point by point and HP is designed to obtain the column vector spatial features \(F_{space}\in R^{C_{1}\times K}\) by dividing the feature map into \(K\) parts according to the height. The HP process can be formulated as Equation (6):  
\[F_{spa}=maxpool\big{(}F_{sp}\big{)}+avgpool\big{(}F_{sp}\big{)} \tag{6}\]  
where \(maxpool\) represents the horizontal maximum pooling and \(avgpool\) represents the horizontal average pooling.