# Non-Local Spatial and Temporal Attention Network for Video-Based Person Re-Identification
## 3 Our Approach
### Temporal Aggregation  
In video-based person Re-ID, a key challenge is how to combine frame-level features into video-level features to express the temporal features in the video better. In previous works, researchers generally used temporal pooling to perform temporal feature aggregation. Table 1 makes a detailed comparison of three different temporal aggregation layers (TPavg, TA, RNN), from which we can see that temporal pooling shows the best performance indicators. However, temporal pooling naturally ignores the temporal relationship between frames. The calculation of each attention score in non-local attention depends on all the frames in the sequence, so it can capture the remote dependencies in the deep neural network well. In this part, we propose a non-local temporal pooling (NLTP) method for temporal feature aggregation. The specific architecture is shown in Figure 4. Our proposed method aims to use the non-local attention mechanism to frame-level feature sequences in the temporal dimension. Our proposed method aims to use the non-local attention mechanism for the extracted frame-level features to perform feature aggregation in the temporal dimension. Enhancing the frame-to-frame relationship allows frames with a closer relevance to obtain a higher weight, resulting in a more reliable person Re-ID model. The NLTP operation we proposed is as follows:  
\[F^{\prime}=\text{Avg}[W(Y)+F], \tag{4}\]  
where \(F\) represents the frame-level features of a sequence extracted through Section 3.1. Equation (4) represents the entire NLTP process, and the result \(Y\) after the non-local operation is restored to the same scale as the input \(F\) through \(W\) (\(1\times 1\) Conv) and elementwise addition is performed with \(F\). Finally, the average pooling operation is performed in the temporal dimension to obtain the final output result \(F^{\prime}\) of NLTP. We assume that a continuous image sequence contains N frames, and \(f_{i}(i\in\{1,2,\dots,N\})\) is one frame of N frames of the video. We use the function \(\text{h}\big{(}x_{i},x_{j}\big{)}\in\mathbb{R}^{N\times N}\) to calculate the scale value between the current frame \(x_{i}\) and each frame \(x_{j}(j\in\{1,2,\dots,N\})\). There are many different choices for the pairwise function \(\text{h}\big{(}x_{i},x_{j}\big{)}\)[14]. In our work, we use the \"dot product\" pairwise function to  
Figure 4: Description of non-local temporal attention (NLTP). The size of the input feature maps F is \(\text{N}\times\text{C}\times\text{H}\times\text{W}\) (N represents the number of frames included in the input video, C is the number of channels of each frame feature map, and H and W indicate the height and width of the input feature map, respectively). The symbol ‘T’ stands for the transpose function. ‘\(1/N\)’ is used for normalization. ‘\(\oplus\)’ denotes the addition operation based on elementwise. ‘\(\otimes\)’ denotes matrix multiplication. The yellow box defines four different \(1\times 1\) convolution blocks. We used the dot product method to calculate the attention score and generate an N\(\times\)N attention matrix. compute the correlation between frames. Recall that each frame is represented as a \(\text{C}\times\text{H}\times\text{W}\) tensor (see Section 3.1). We apply a \(1\times 1\) convolution on \(f_{i}\) to reduce its channel dimension to \(C^{\prime}=C/2\) as a way to reduce computation. Then, \(f_{i}\) is reshaped to a vector. We use \(\theta(f_{i})\), \(\varphi\big{(}f_{j}\big{)}\), and \(g\big{(}f_{j}\big{)}\) to indicate three such vectors. The pairwise function is then defined as the point product between \(\theta(x_{i})\) and \(\varphi\big{(}x_{j}\big{)}\). The pairwise function is defined as:  
\[h_{(f_{i},f_{j})}=\theta(x_{i})^{T}\varphi\big{(}x_{j}\big{)}. \tag{5}\]  
Then, we multiply the output of the pairwise function by \(1/N\) as the normalization operation. We call the normalized result the \"attention score\" to indicate the influence of all frames \(f_{j}\) on \(f_{i}\).