# Multiple Visual-Semantic Embedding for Video Retrieval from Query Sentence
## 3 the Proposed Method
### Sequential Visual Network  
Similar to the global visual network, we divide the input video frames into \(N\) chunks and take the first frames of each chunk as the input of the sequential visual network. Thus, we use 20 discontinuous frames. Then, we use the ResNet-152 to extract a \(7\times 7\times 2048\)-dimensional vector \(\psi_{s}(x)\) from each frame at the last convolution layer of the ResNet. The \(\psi_{s}(x)\) contains visual features in spatial regions. Considering spatial regions where we should pay further attention may change by sentences, we need to explore relationships between spatial and textual features. Therefore, we incorporate a spatial attention mechanism into the sequential visual network. We apply the attention mechanism to \(\psi_{s}(x)\) to emphasize spatial regions. Finally, we capture the sequential information of the video using a single layer Long-Short Term Memory (LSTM) [36] with an \(H\)-dimensional hidden state. We denote the vector embedded by the sequential visual network as \(\mathrm{f}_{s}(x)\in\mathbb{R}^{D}\).  
The sequential visual network uses LSTM to capture a meaningful sequential representation of the video. However, the LSTM transforms all spatial details of a video into a flat representation, resulting in losing its spatial reasoning with the sentence. Therefore, we employ the spatial attention mechanism in order to obtain a spatial relationship between video and sentence. Inspired by the work [18], we develop the spatial attention mechanism to learn with regions in a frame to attend to each word in the sentence.  
We illustrate the spatial attention mechanism in Figure 2. The mechanism associates the visual feature vector \(\psi_{s}(x)\) with the textual vector \(\phi(y)\) to produce a spatial attention map \(a\in\mathbb{R}^{7\times 7}\). Then, we combine \(\psi_{s}(x)\) with \(a\) to produce the spatial attended feature \(\psi_{a}(x)\). Formally, \(\psi_{a}(x)\) and \(a\) are defined in Equation (5) and Equation (6), respectively.  
Specifically, Equation (4) defines an embedding function of the sequential visual network. \(\gamma\) represents the LSTM, and \(\otimes\) is the element-wise product. The \(p\) and \(q\) mean intermediate outputs, which are 512-dimensional vectors. We have learnable parameters \(W_{a}\in\mathbb{R}^{512\times(7\times 7)}\), \(W_{p}\in\mathbb{R}^{(7\times 7\times 2048)\times 512}\), \(\hat{W}_{q}\in\mathbb{R}^{H\times 512}\), \(b_{a}\in\mathbb{R}^{7\times 7}\), and \(b_{p},\hat{b}_{q}\in\mathbb{R}^{512}\).  
\[\mathsf{f}_{\mathsf{s}}(x) =\gamma(\psi_{a}(x)) \tag{4}\] \[\psi_{a}(x) =\psi_{s}(x)\otimes a\] (5) \[a =\text{softmax}(\tanh(W_{a}(p+q)+b_{a}))\] (6) \[p =\tanh(W_{p}\psi_{s}(x)+b_{p})\] (7) \[q =\tanh(\hat{W}_{q}\phi(y)+\hat{b}_{q}) \tag{8}\]  
The joint embedding space of the sequential visual and textual networks is formed by \(\mathsf{f}_{\mathsf{s}}(x)\) and \(\mathsf{g}_{\mathsf{s}}(y)\). We measure the similarity \(\mathsf{s}_{\mathsf{s}}(x,y)\) in this joint embedding space. The formulations are described below, where \(\hat{W}_{\mathsf{s}}\in\mathbb{R}^{H\times D}\) and \(\hat{b}_{s}\in\mathbb{R}^{D}\) are learnable parameters.  
\[\mathsf{g}_{\mathsf{s}}(y) =\hat{W}_{s}\phi(y)+\hat{b}_{s} \tag{9}\] \[\mathsf{s}_{\mathsf{s}}(y,x) =\frac{\mathsf{f}_{\mathsf{s}}(x)\cdot\mathsf{g}_{\mathsf{s}}(y )}{\parallel\mathsf{f}_{\mathsf{s}}(x)\parallel\parallel\mathsf{g}_{\mathsf{ s}}(y)\parallel} \tag{10}\]