# Non-Local Spatial and Temporal Attention Network for Video-Based Person Re-Identification
## 3 Our Approach
### Frame-Level Feature Extraction  
In the process of frame-level feature extraction, we use the most modern image recognition network architecture SE-ResNet50 as a feature extractor in video-based Re-ID. SE-ResNet50 contains five consecutive CNN blocks (one initial convolution block, followed by four successive squeeze-and-excitation (SE) residual blocks). We argue that a single CNN is not sufficient for feature extraction of an image. With the addition of an attentional mechanism, CNN can extract more critical spatial information from the image, similar to human visual attention. We add NLAM between CNN blocks to obtain a better frame-level feature representation. The overall structure of NLAM is shown in Figure 3.  
Figure 3: Non-local attention module (NLAM) inserted between the \(L^{th}\) and \(L+1^{th}\) blocks of CNN. NLAM mainly includes two parts: (**a**) NLAM spatial attention; (**b**) NLAM channel attention. X represents the feature maps output by the \(L^{th}\) CNN block; \(A_{5}\) expresses the feature maps after NLAM spatial attention; \(A_{C}\) indicates the feature maps after NLAM channel attention as the input of the \(L+1^{th}\) block. ‘\(\varphi\)’ denotes the addition operation based on elementwise. ‘\(\varphi\)’ denotes matrix multiplication. The green box represents \(1\times 1\) convolution. GMP = Global Maximum Pooling, GAP = Global Average Pooling, MLP = Multi-Layer Perceptron. Figure 2: Our overall network architecture. \(X\) means the input video; \(F\) represents the frame-level features after passing through the feature extractor; \(F^{\prime}\) refers to the resulting video-level representation. NLAM = Non-Local Attention Module; NLTP = Non-Local Temporal Pooling; FC = Fully Connected layer. In the NLAM spatial attention part, we aim to perform spatial attention calculation on the feature maps output by the CNN network in the previous layer. Given an input feature tensor \(\text{X}\in\mathbb{R}^{N\times\text{C}\times H\times W}\), it is obtained from a sequence of \(N\) feature maps of size \(\text{C}\times\text{H}\times\text{W}\). We aim to exchange their spatial information between all frames in the sequence to determine a better position of the target person in the image and reduce the interference of occlusion between frames. Let \(x_{i}\) be sampled from \(\text{X}\). First, we reduce the dimension of the input feature channel to \(C^{\prime}\) through three \(1\times 1\) convolution blocks (a, b, d) to obtain \(\text{A},\text{ B},\text{ D}\in\mathbb{R}^{C^{\prime}\times NHW}\). The transposition of A is multiplied by B to obtain the attention score of all positions \(x_{j}\) at position \(x_{i}\) by using embedded Gaussian instantiation. Then, the weighted average \(\text{M}\in\mathbb{R}^{NHW\times NHW}\) of the attention scores of all positions \(x_{j}\) is used to calculate the response \(y_{i}\) of each position \(x_{i}\). Finally, Y is recovered to the same size as the input X by \(1\times 1\) convolution, and the recovered result is added to the original feature tensor X to obtain the final result \(A_{S}\). NLAM spatial attention can be formulated as follow:  
\[y_{i}=\frac{1}{\sum_{\forall j}e^{\sigma(x_{i})^{T}b(x_{j})}}\sum_{\forall j}e^ {\sigma(x_{i})^{T}b(x_{j})}d\Big{(}x_{j}\Big{)}, \tag{1}\]  
\[A_{S}=W_{0}(\text{Y})+\text{X}. \tag{2}\]  
Equation (1) represents the process of non-local operations, and the overall spatial attention is formulated as Equation (2). Here, \(\text{i},\text{j}=[1,\,NHW]\) refers to all locations of each feature map and in all frames. The convolution operation is expressed by a, b, and c. \(W_{0}\) recovers Y to the same size as the input tensor X. The idea contained in the non-local operation is that when extracting features at a specific location in a particular time, the network should consider the spatial and temporal dependencies within the sequence by attending on the non-local context.