# An Efficient Person Search Method Using Spatio-Temporal Features for Surveillance Videos +
## 3 The Proposed Method
### Spatio-Temporal Feature Generation  
Surveillance video is not only spatially invariant but also temporally continuous. To organize the temporal relationship of persons between adjacent frames, Gated Recurrent Unit (GRU) [42] is employed to process the spatial features of persons extracted from each frame. GRU is a type of Recurrent Neural Network (RNN) and solves the vanishing gradient problem in a standard RNN. Derived from Long-Short Term Memory (LSTM) [43], GRU combines a forget gate and an input gate into one single update gate and utilizes the hidden state to transfer information. Compared with LSTM, the parameters of GRU are fewer and the structure of GRU is relatively simple. Thus, GRU can be trained more easily than LSTM. Because GRU can process data sequences efficiently, it is used to model the temporal correlation of the same person between adjacent frames. In Figure 2, the spatial feature \(s_{i,j}\) is taken as an input vector and fed into GRU. Firstly, reset gate \(c_{i,j}\) and update gate \(d_{i,j}\) can be computed as:  
\[c_{i,j}=\sigma(W_{c}s_{i,j}+U_{c}h_{i-1,j}+b_{c}) \tag{2}\]  
\[d_{i,j}=\sigma(W_{d}s_{i,j}+U_{d}h_{i-1,j}+b_{d}) \tag{3}\]  
where \(h_{i-1,j}\) is the hidden state of the \(j\)-th person in the frame \(v_{i-1}\), \(\sigma(x)=\left(1+e^{-x}\right)^{-1}\) is the sigmoid non-linearity function that maps real-value inputs into the interval \([0,1]\), \(W_{c}\), \(U_{c}\), and \(b_{c}\) are the parameters of reset gates \(c_{i,j}\), and \(W_{d}\), \(U_{d}\) and \(b_{d}\) are the parameters of update gates \(d_{i,j}\). Figure 2: The structure of GRU. Next, in a new gate, the candidate hidden state \(h_{i,j}^{\prime}\) is updated by spatial feature \(s_{i,j}\), reset gate \(c_{i,j}\), and hidden state \(h_{i-1,j}\), and is computed as:  
\[h_{i,j}^{\prime}=\tanh(W_{h}s_{i,j}+U_{h}(c_{i,j}\odot h_{i-1,j})+b_{h}) \tag{4}\]  
where \(\tanh\) is activation function, \(\odot\) represents the element-wise product, and \(W_{h}\), \(U_{h}\) and \(b_{h}\) are gate parameters. Finally, the hidden state \(h_{i,j}\) is generated by update gate \(d_{i,j}\), hidden state \(h_{i-1,j}\), and candidate hidden state \(h_{i,j}^{\prime}\), and is computed as:  
\[h_{i,j}=d_{i,j}\odot h_{i-1,j}+(1-d_{i,j})\odot h_{i,j}^{\prime} \tag{5}\]  
from Equation (5), the hidden state \(h_{i,j}\) not only considers the candidate hidden state \(h_{i,j}^{\prime}\) in the frame \(v_{i}\), but also involves the hidden state \(h_{i-1,j}\) in the frame \(v_{i-1}\), thereby establishing the temporal relationship of the \(j\)-th person between frame \(v_{i-1}\) and \(v_{i}\). After all the spatial features of persons have been processed by GRU, for the \(j\)-th person, we can obtain a corresponding hidden state sequence \(h_{j}=\left\{h_{i,j}\right\}(1\leq i\leq n_{j})\), which represents temporal correlation of persons in the \(n_{j}\) frames. Then, all the persons in the video can be represented by a collection of hidden state sequences \(H=\left\{h_{j}\right\}(1\leq j\leq M)\). The hidden state \(h_{i,j}\) is a 256-dimensional vector, thus the dimension of the hidden state sequence \(h_{j}\) is \(256\times n_{j}\), and the dimension of hidden state sequences \(H\) is \(256\times n_{j}\times M\). Due to the high dimensionality of \(h_{j}\) and \(H\), the computational cost is increased by directly utilizing the hidden state sequences in the search process. To reduce the dimension of the hidden states and generate spatio-temporal features for each person, an average pooling layer is added to the output of GRU. After averaging the hidden state sequence \(h_{j}\), for the \(j\)-th person, the spatio-temporal feature can be represented as:  
\[p_{j}=\frac{1}{n_{j}}\sum_{i=1}^{n_{j}}h_{i,j} \tag{6}\]  
from Equation (6), the average pooling layer finally transforms a \(256\times n_{j}\) dimensional hidden state sequence \(h_{j}\) into a 256-dimensional spatio-temporal feature \(p_{j}\), thereby reducing the computational cost.