# Multiple Visual-Semantic Embedding for Video Retrieval from Query Sentence
## 3 the Proposed Method  
### Overview  
We describe the problem of learning a joint embedding space for video and sentence embedding. Given video and sentence instances, which are sequences of frames and words, the aim is to train embedding functions that map them into a joint embedding space. Formally, we use embedding functions \(\mathsf{f}:\mathcal{X}\rightarrow\mathcal{Z}\) and \(\mathsf{g}:\mathcal{Y}\rightarrow\mathcal{Z}\), where \(\mathcal{X}\) and \(\mathcal{Y}\) are video and sentence domains, respectively, and \(\mathcal{Z}\) is a joint embedding space. The similarity \(\mathsf{s}(\mathsf{f}(x),\mathsf{g}(y))\) between \(\mathcal{X}\) and \(\mathcal{Y}\) is calculated in \(\mathcal{Z}\) by a certain measurement. Therefore, the ultimate goal is learning \(\mathsf{f}\) and \(\mathsf{g}\) satisfying the following equation: \(\mathsf{s}(\mathsf{f}(x_{i}),\mathsf{g}(y_{i}))>\mathsf{s}(\mathsf{f}(x_{i}), \mathsf{g}(y_{i})),\forall\;i\
eq j\). This encourages similarity to increase in a same pair \(x_{i}\) and \(y_{i}\), whereas it decreases in a different pair \(x_{i}\) and \(y_{j}\).  
As we illustrated in the overview of the proposed framework in Figure 1, there are two networks for embedding videos: global and the sequential visual networks. These networks have their counterparts that are embedding the sentence. Thus, we develop multiple visual and textual networks that are responsible for embedding a video and a sentence, respectively. We form one embedding space by merging two networks (one visual and one textual) so that we can align a visual feature to textual information. Specifically, the global visual network aligns average visual information of the video to the sentence. Likewise, the sequential visual network aligns a temporal context to the sentence. Consequently, the networks receive a video and sentence as inputs and map them into the joint embedding spaces.  
We propose to fuse similarities measured in the embedding spaces to produce the final similarity. One embedding space has the global visual network extracting a visual object feature from the video. Thus, this embedding space describes a relationship between global visual appearances and textual features. The other embedding space captures the relationship between sequential appearances and textual features. The similarity scores of the two embedding spaces are then combined using the weighted sum to put more emphasis on one embedding space than the other space.  
Our motivation is twofold. Firstly, we aim to develop a more robust similarity measurement by combining multiple embedding spaces in a weighted manner instead of using a hard-coded average. Videos and sentences require flexible attention, such as attention to global and local visual features. For example, the words in \"a girl is dancing\" need different visual feature perspectives. \"Girl\" needs to attend global visual features. In contrast, \"dancing\" needs to attend local visual features to capture temporal information. Secondly, in order to highlight spatial and temporal information of videos according to a sentence, we need to develop a mechanism that utilizes textual information to emphasize spatial and temporal features. Overall, we can dynamically attend to global and local visual features by varying their weights. Consequently, we can learn appropriate correspondences between visual and textual features.