Learning Joint Representations of Videos andSentences with Web Image SearchMayu Otani1 , Yuta Nakashima1 , Esa Rahtu2 ,Janne Heikkilä2 , and Naokazu Yokoya1
1

Graduate School of Information Science, Nara Institute of Science and Technology{otani.mayu.ob9,n-yuta,yokoya}@is.naist.jp
2Center for Machine Vision and Signal Analysis, University of Oulu
{erahtu,jth}@ee.oulu.fi

Abstract. Our objective is video retrieval based on natural language
queries. In addition, we consider the analogous problem of retrieving
sentences or generating descriptions given an input video. Recent work
has addressed the problem by embedding visual and textual inputs into
a common space where semantic similarities correlate to distances. We
also adopt the embedding approach, and make the following contributions: First, we utilize web image search in sentence embedding process
to disambiguate fine-grained visual concepts. Second, we propose embedding models for sentence, image, and video inputs whose parameters
are learned simultaneously. Finally, we show how the proposed model
can be applied to description generation. Overall, we observe a clear improvement over the state-of-the-art methods in the video and sentence
retrieval tasks. In description generation, the performance level is comparable to the current state-of-the-art, although our embeddings weretrained for the retrieval tasks.
Keywords: Video retrieval, sentence retrieval, multimodal embedding,neural network, image search, representation learning

1

Introduction

During the last decade, the Internet has become an increasingly important distribution channel for videos. Video hosting services like YouTube, Flickr, andVimeo have millions of users uploading and watching content every day. At thesame time, powerful search methods have become essential to make good useof such vast databases. By analogy, without textual search tools like Google or
Bing, it would be nearly hopeless to find information from the websites.
Our objective is to study the problem of retrieving video clips from a databaseusing natural language queries. In addition, we consider the analogous problem
of retrieving sentences or generating descriptions based on a given video clip. We
are particularly interested in learning appropriate representations for both visualand textual inputs. Moreover, we intend to leverage the supporting information
provided by the current image search approaches.

2

M. Otani et al.
Sentence“A baby is playing a guitar.”

Sentence
EmbeddingVideo
EmbeddingVideo

Web Images

Image Search
Embedding Space

Fig. 1. An overview of our approach. Left side illustrates the image search results fora query “A baby is playing a guitar”. Images highlight evidence of objects (“baby”,“guitar”) and actions (“playing”). Right side shows the most relevant videos in theYouTube dataset [1] obtained by ranking the clips according to Euclidean distance tothe query sentence in an embedding space.

This topic has recently received plenty of attention in the community, andpapers have presented various approaches to associate visual and textual data.
One direction to address this problem is to utilize metadata that can be directlycompared with queries. For instance, many web image search engines evaluate
the relevance of an image based on the similarity of the query sentence with theuser tags or the surrounding HTML text [4]. For sentence retrieval, Ordonez et
al. [21] proposed to compare an image query and visual metadata with sentences.
While these methods using comparable metadata have demonstrated impressive results, they do not perform well in cases where appropriate metadata islimited or not available. Moreover, they rely strongly on the assumption thatthe associated visual and textual data in the database is relevant to each other.
These problems are more apparent in the video retrieval task since video distribution portals like YouTube often provide less textual descriptions compared toother web pages. Furthermore, available descriptions (e.g. title) often cover onlya small portion of the entire visual content in a video.
An alternative approach would be to compare textual and visual inputs directly. In many approaches, this is enabled by embedding the corresponding
representations into a common vector space in such a way that the semantic
similarity of the original inputs would be directly reflected in their distance inthe embedding space (Fig. 1). Recent work [27, 16] has proposed deep neuralnetwork models for performing such embeddings. The results are promising, butdeveloping powerful joint representations still remains a challenge.
In this paper, we propose a new embedding approach for sentence and video
inputs that combines the advantages of the metadata-based web image searchand deep neural network-based representation learning. More precisely, we use astandard search engine to obtain a set of supplementary images for each query
sentence. Then, we pass the sentence and the retrieved images to a two-branch

Learning Joint Representations of Videos and Sentences3

neural network that produces the sentence embedding. The video inputs areembedded into the same space using another neural network. The network parameters are trained jointly so that videos and sentences with similar semantic
content are mapped to close points. Figure 1 illustrates the overall architectureof our approach. The experiments indicate a clear improvement over the current
state-of-the-art baseline methods.
Our main contributions are as follows:– We present an embedding approach for video retrieval that incorporates web
image search results to disambiguate fine-grained visual concepts in query
sentences.
– We introduce neural network-based embedding models for video, sentence,and image inputs whose parameters can be learned jointly. Unlike previous
work that uses only videos and sentences, we utilize a sentence and corresponding web images to compute the sentence embedding.
– We demonstrate a clear improvement over the state-of-the-art in the video
and sentence retrieval tasks with the YouTube dataset [1].
– We demonstrate description generation as an example of possible applications of our video embeddings. We observed that the performance is comparable with the state-of-the-art. This indicates that video contents are efficiently encoded into our video embeddings.

2

Related Work

Visual and Language Retrieval: Due to the explosive growth of images andvideos on the web, visual retrieval has become a hot topic in computer vision andmachine learning [4, 20]. Several recent approaches for joint representation leaning enable direct comparison among different multimodalities. Farhadi et al. [7]introduced triplets of labels on object, action, and scene as joint representations
for images and sentences. Socher et al. [27] proposed to embed representationsof images and labels into a common embedding space. For videos, the approach
proposed by Lin et al. [18] associates a parsed semantic graph of a query sentence
and visual cues based on object detection and tracking.
The recent success of deep convolutional neural networks (CNNs) together
with large-scale visual datasets [22, 2, 25] has resulted in several powerful representation models for images [5, 33, 35]. These CNN-based methods have beensuccessfully applied to various types of computer vision tasks, such as object
detection [10, 23], video summarization [12], and image description generation[32, 6].
Deep neural networks have also been used in the field of natural language processing [17, 16]. For example, Kiros et al. [16] proposed sentence representation
learning based on recurrent neural networks (RNNs). They also demonstrated
image and sentence retrieval by matching sentence and image representations
with jointly leaned linear transformations.
Representation learning using deep neural networks is explored in many tasks[3, 19, 9, 14, 34, 37]. Frome et al. [9] proposed image classification by computing

4

M. Otani et al.
Sentence
“A”

“dog”

“is”

“eating” “watermelon”

RNN

RNN

RNN
RNN
RNN
RNN for Sentences“.”
RNN
Pooling+}

Web images

Loss

CNN for Web Images
Pooling}

Video

CNN for Videos

Fully-connected Layers

Fig. 2. Illustration of our video and sentence embedding. The orange component is
the sentence embedding model that takes a sentence and corresponding web images as
input. Video embedding model is denoted by the blue component.

similarity between joint representations of images and labels, and Zhu et al. [37]addressed alignment of a movie and sentences in a book using joint representations for video clips and sentences. Their approach also computes similarity
between sentences and subtitles of video clips to improve the alignment of video
clips and sentences.
Our approach is the closest to work by Xu et al. [34]. They represent a
sentence by a subject, verb, and object (SVO) triplet, and embed sentences aswell as videos to a common vector space using deep neural networks. The maindifference between ours and the work [34] is the use of an RNN to encode asentence and supplementary web images. The use of an RNN enables our model
to encode all words in a sentence and capture details of the sentence, such as an
object’s attributes and scenes, together with corresponding web images.
Exploiting Image Search: The idea of exploiting web image search is adopted
in many tasks, including object classification [8] and video summarization [28].
These approaches collect a vast amount of images from the web and utilize
them to extract canonical visual concepts. Recent label prediction for images
by Johnson et al. [13] infers tags of target images by mining relevant Flickr
images based on their metadata, such as user tags and photo groups curated
by users. The relevant images serve as priors on tags for the target image. A
similar motivation drives us to utilize web images for each sentence, which candisambiguate visual concepts of the sentence and highlight relevant target videos.

3

Proposed Approach

We propose neural network-based embedding models for the video and sentence
retrieval tasks. In order to enhance the sentence embedding, we retrieve relevantLearning Joint Representations of Videos and Sentences5

web images that are assumed to disambiguate semantics of the sentence. For
example, the word “keyboard” can be interpreted as a musical instrument or
an input device for computers. If the word comes with “play,” the meaning of“keyboard” narrows down to a musical instrument. This means that a specificcombination of words can reduce the possible visual concepts relevant to thesentence, which may not be fully encoded even with the state-of-the-art RNNbased approach like [16].
We propose to take this into account by using web image search results.
Since most image search engines use surrounding text to retrieve images, we
can expect that they are responsive to such word combinations. Consequently,
we retrieve web images using the input sentence as a query and download theresults. The web images are fused with the input sentence by applying a twobranch neural network as shown in Fig. 2. Videos are also encoded by applying
a neural network-based video embedding model. Relevance between sentence
and video inputs is directly computed in the common embedding space using
Euclidean distances. We jointly train our embedding models using video-sentencepairs by minimizing the contrastive loss [3].
3.1Video Embedding

We extract frames from a video at 1 fps as in [34] and feed them to a CNN-based
video embedding model. In our approach, we employ two CNN architectures: 19layer VGG [26] and GoogLeNet [29], both of which are pre-trained on ImageNet[25]. We replace the classifier layer in each model with two fully-connected layers.
Specifically, we compute activations of the VGG’s fc7 layer or the GoogLeNet’s
inception 5b layer and feed them to additional embedding layers.
Let X = {xi | i = 1, . . . , M } be a set of frames xi , and CNN(xi ) ∈ Rdv bean activation of a CNN (dv =4,096 for VGG, and dv =1,024 for GoogLeNet). The
video embedding φv (X) ∈ Rde is computed by:
1 X
φv (X) =tanh(Wv2 tanh(Wv1 CNN(xi ) + bv1 ) + bv2 ).
(1)
M
xi ∈X

Here, Wv1 ∈ Rdh ×dv , bv1 ∈ Rdh , Wv2 ∈ Rde ×dh , and bv2 ∈ Rde are the learnable
parameters of the fully-connected layers.
3.2

Sentence and Web Image Embedding

The sentence embedding model consists of two branches that merge the outputsof a CNN-based network for web images and an RNN-based network for a sentence. Before computing the sentence embedding, we download top-K results of
web image search with the input sentence as a query. Let Z = {zj | j = 1, . . . , K}
be a set of web images. We utilize the same architecture as the video embedding
and compute an intermediate representation ez ∈ Rde that integrates the web
images as:
1 X
ez =tanh(Wz2 tanh(Wz1 CNN(zj ) + bz1 ) + bz2 ),
(2)
K
zj ∈Z

6

M. Otani et al.

where Wz1 ∈ Rdh ×dv , bz1 ∈ Rdh , Wz2 ∈ Rde ×dh , and bz2 ∈ Rde are the leanable
parameters of the two fully-connected layers.
We encode sentences into vector representations using skip-thought that is
an RNN pre-trained with a large-scale book corpus [16]. Let Y = {yt | t =1, . . . , TY } be the input sentence, where yt is the t-th word in the sentence, and
TY is the number of words in the sentence Y . Skip-thought takes a sequence ofword vectors wt ∈ Rdw computed from a word input yt as in [16] and produces
hidden state ht ∈ Rds at each time step t as:
rt = σ(Wr wt + Ur wt−1 ),(3)

it = σ(Wi wt + Ui ht−1 ),(4)

at = tanh(Wa wt + Ua (rtht−1 )),(5)ht = (1 − it )

at ,

(6)

ht−1 + it

where σ is the sigmoid activation function, and is the component-wise product.
The parameters Wr , Wi , Wa , Ur , Ui , and Ua are ds × dw matrices. Sentence Y is
encoded into the hidden state after processing the last word wTY , i.e., hTY . We
use combine-skip in [16], which is a concatenation of outputs from two separate
RNNs trained with different datasets. We denote the output of combine-skipfrom sentence Y by sY ∈ Rdc , where dc =4,800.
We also compute an intermediate representation es for sentence Y as:
es = tanh(Ws2 tanh(Ws1 sY + bs1 ) + bs2 ),(7)

where Ws1 ∈ Rdh ×dc , bs1 ∈ Rdh , Ws2 ∈ Rde ×dh , and bs2 ∈ Rde are the learnable
parameters of sentence embedding.
Once the outputs es and ez of each branch in our sentence embedding model
are computed, they are merged into a sentence embedding φs (Y, Z) as:φs (Y, Z) =1
(es + ez ).
2(8)

By this simple mixture of es and ez , the sentence and web images directly influence the sentence embedding.
3.3

Joint Learning of Embedding Models

We jointly train both embedding φv and φs using pairs of videos and associatedsentences in a training set by minimizing the contrastive loss function [3]. In
our approach, the contrastive loss decreases when embeddings of videos andsentences with similar semantics get closer to each other in the embedding space,
and those with dissimilar semantics get farther apart.
The training process requires a set of positive and negative video-sentencepairs. A positive pair contains a video and a sentence that are semantically relevant, and a negative pair contains irrelevant ones. Let {(Xn , Yn ) | n = 1, . . . , N }be the set of positive pairs. Given a positive pair (Xn , Yn ), we sample irrelevant
sentences Yn0 = {Yf0 | f = 1, . . . , Nc } and videos Xn0 = {Xg0 | g = 1, . . . , Nc } from

7

count

count

Learning Joint Representations of Videos and Sentences

distance

count

count

distance

distance

distance

Fig. 3. Histograms of pairwise distances before training (left) and after training (right).
Top row: Histograms of the training set. Bottom row: Histograms of the test set. Red
represents positive pairs, and green represents negative pairs.

the training set, which are used to build two sets of negative pairs {(Xn , Yf0 ) |Yf0 ∈ Yn0 } and {(Xg0 , Yn ) | Xg0 ∈ Xn0 }. In our approach, we set the size of negative
pairs Nc to 50. We train the parameters of embedding φv and φs by minimizing
the contrastive loss defined as:1
Loss(Xn , Yn ) = 1+2Nc d(Xn , Yn )X+
max(0, α − d(Xn , Yf0 ))0
Yf0 ∈Yn+X
max(0, α − d(Xg0 , Yn )) ,(9)

Xg0 ∈Xn0d(Xi , Yj )=||φv (Xi ) − φs (Yj , Zj )||22 ,(10)

where Zn is the web images corresponding to sentence Yn . The hyperparameter α
is a margin. Negative pairs with smaller distances than α are penalized. Margin
α is set to the largest distance of positive pairs before training so that mostnegative pairs influence the model parameters at the beginning of training.
Figure 3 shows the histograms of distances of positive and negative pairs before and after training. The initial distance distributions of positive and negative
pairs overlap. After training, the distributions are pulled apart. This indicatesthat the training process encourages videos and sentences in positive pairs to bemapped to closer points and those in negative ones to farther points.
The examples of positive and negative pairs in our test set with corresponding
distances are shown in Fig. 4. The positive pair (a) and (b) are easy cases, inwhich sentences explicitly describe the video contents. The pair (c) is an exampleof hard cases. The sentence includes “a man” and “phone”, but the video actuallyshows two men, and a phone is occluded by a hand.

8

M. Otani et al.
(a) A hamster is eating seeds.

(b) A man is playing guitar.

(c) A man is talking on a phone.

(7.26)(12.78)(4.52)(6.54)(d) A woman is mixing a yolk
into a dough mixture.

(8.50)(e) A person puts someground beef in a pan.

(15.69)(f) A woman is slicing a carrot.

Fig. 4. Examples of positive (a)–(c) and negative (d)–(f) pairs in the test set withcorresponding distances. The values (·) are distances of the pairs. The plot is thehistograms of distances of positive (red) and negative (green) pairs.

The pairs (d) and (e) are hard negative cases. The pair (d) shows partial
matches of contents, such as the action “mixing” and the object “yolk.” Anothernegative pair (e) has a video and a sentence about cooking, although there isdisagreement about details. As shown in these examples, the closer a video anda sentence are located in the embedding space, the more relevant they are. More
examples can be found in the supplementary material.

4

Retrieval Experiments

4.1

Implementation DetailWith 19-layer VGG, the hidden layer size dh of embedding φv and φs was set to
1,000 and the dimension of the embedding space de was set to 300. For modelusing GoogLeNet, we used dh = 600 and de = 300.
We implemented our model using Chainer [30]. We used Adam [15] for optimization with a learning rate of 1 × 10−4 . The parameters of the CNNs andskip-thought were fixed. We applied dropout with a ratio of 0.5 to the input of
the first and second layers of φv and φs . Our models were trained for 15 epochs,
and their parameters were saved at every 100 updates. We took the model parameters whose performance was the best on the validation set.

Learning Joint Representations of Videos and Sentences

4.2

9

Experimental Setup

Dataset: We used the YouTube dataset [1] consisting of 80K English descriptions for 1,970 videos. We first divided the dataset into 1,200, 100, and 670
videos for training, validation, and test, respectively, as in [35, 34, 11]. Then, we
extracted five-second clips from each original video in a sliding-window manner.
As a result, we obtained 8,001, 628, and 4,499 clips for the training, validation,and test sets, respectively. For each clip, we picked five ground truth descriptions
out of those associated with its original video.
We collected top-5 image search results for each sentence using the Bing image search engine. We used a sentence modified by lowercasing and punctuation
removal as a query. In order to eliminate cartoons and clip art, the image type
was limited to photos using Bing API.

Video Retrieval: Given a video and a query sentence, we extracted five-second
video clips from the video and computed Euclidean distances from the query to
the clips. We used their median as the distance of the original video and thequery. We ranked the videos based on the distance to each query and recorded therank of the ground truth video. Since the test set has 670 videos, the probabilityof bringing the ground truth video at top-1 by random ranking is about 0.14%.

Sentence Retrieval: For the sentence retrieval task, we ranked sentences foreach query video. We computed the distances between a sentence and a queryvideo in the same way as the video retrieval task. Note that each video has fiveground truth sentences; thus, we recorded the highest rank among them. The
test set has 3,500 sentences.

Evaluation Metrics: We report recall rates at top-1, -5, and -10, the averageand median rank, which are standard metrics employed in the retrieval evaluation. We found that some videos in the dataset had sentences whose semantics
were almost the same (e.g., “A group of women is dancing” and “Women are
dancing”). For the video that is annotated with one of such sentences, the othersentence is treated as incorrect with the recall rates, which does not agree withhuman judges. Therefore, we employed additional evaluation metrics widely usedin the description generation task, i.e., CIDEr, BLUE@4, and METEOR [2].
They compute agreement scores in different ways using a retrieved sentence anda set of ground truth ones associated with a query video. Thus, these metrics givehigh scores for semantically relevant sentences even if they are not annotated toa query video. We computed the scores of the top ranked sentence for each videousing the evaluation script provided in the Microsoft COCO Evaluation Server[2]. In our experiments, all ground truth descriptions for each original video areused to compute these scores.

10

M. Otani et al.

Table 1. Video and sentence retrieval results. R@K is recall at top K results (higher
values are better). aR and mR are the average and median of rank (lower values are
better). Bold values denotes best scores of each metric.
Video retrieval
Sentence retrieval
ModelsR@1 R@5 R@10 aR mR R@1 R@5 R@10 aR mR
Random Ranking 0.14 0.79 1.48 335.92 333 0.22 0.69 1.32 561.32 439
VGG+VS6.12 21.88 33.22 58.98 24 7.01 18.66 27.16 131.33 35
VGG+VI
4.03 13.70 21.40 94.62 48 5.67 17.91 28.21 116.86 38
VGG+ALL1
6.48 20.15 30.51 59.53 26 10.60 25.22 36.42 85.90 21
VGG+ALL2
5.97 21.31 32.54 56.01 24 8.66 22.84 33.13 100.14 29
GoogLeNet+VS7.49 22.84 33.10 54.14 22 8.51 21.34 30.45 114.66 33
GoogLeNet+VI
4.24 16.42 24.96 84.48 41 6.87 17.31 30.00 96.78 30
GoogLeNet+ALL1 5.52 18.93 28.90 60.38 28 9.85 27.01 38.36 75.23 19
GoogLeNet+ALL2 7.67 23.40 34.99 49.08 21 9.85 24.18 33.73 85.16 22
ST [16]
2.63 11.55 19.34 106.00 51 2.99 10.90 17.46 241.00 77
DVCT [34]224.10 236.27 Table 2. Evaluated scores of retrieved sentences. All values are reported in percentage(%). Higher scores are better.
ModelsVGG+VS
VGG+VIVGG+ALL1
VGG+ALL2
GoogLeNet+VS
GoogLeNet+VI
GoogLeNet+ALL1
GoogLeNet+ALL2

4.3

CIDEr
30.44
29.00
42.52
32.56
33.82
35.08
43.52
38.08

BLEU
27.16
22.42
30.81
27.39
26.97
24.56
29.99
29.28

METEOR
25.74
22.99
27.77
26.58
25.99
24.16
27.48
26.50

Effects of Each Component of Our Approach

In order to investigate the influence of each component of our approach, wetested some variations of our full model. The scores of the models on the video
and sentence retrieval tasks are shown in Table 1. Our full model is denoted by
ALL2 . ALL1 is a variation of ALL2 that computes embeddings with one fullyconnected layer with the unit size of de . Comparison between ALL1 and ALL2
indicates that the number of fully-connected layers in embedding is not essential.
In order to evaluate the contributions of web images, we trained a modelthat does not use web images, i.e., an embedding of a sentence Y is computed by φs (Y ) = es . We denote this model by VS. VGG+ALL2 had better
average rank than VGG+VS, and comparison between GoogLeNet+ALL2 and
GoogLeNet+VS also shows a clear advantage of incorporating web images.
We also tested a model without sentences, which is denoted by VI. It computes an embedding of web images by φs (Z) = ez . We investigated the effect ofusing both sentences and web images by comparing VI to our full model ALL2 .

Learning Joint Representations of Videos and Sentences
Query(1) A man is playing a keyboard.

11

GoogLeNet+VS

GoogLeNet+ALL2

GoogLeNet+VI

GoogLeNet+ALL2(2) Kids are playing in a pool.

(3) A man is trimming fat from a roast.

Query(4) A boy is singing into a microphone.

(5) A man shoots a shotgun.

(6) A cat is pawing in a water bowl.

Fig. 5. Examples of video retrieval results. Left: Query sentence and web images. Center: Top-3 retrieved videos by GoogLeNet+VS and VI. Right: Top-3 retrieved videos
by GoogLeNet+ALL2 .

The results show that sentences are necessary. The comparison between VI and
VS also indicates that sentences provide main cues for the retrieval task.
The scores of retrieved sentences computed by CIDEr, BLEU@4, and METEOR are shown in Table 2. In all metrics, our model using both sentences andweb images (ALL1 and ALL2 ) outperformed to other models (VS and VI). In
summary, contributions by sentences and web images were non-trivial, and thebest performance was achieved by using both of them.
Some examples of retrieved videos by GoogLeNet+VS, GoogLeNet+VI, and
GoogLeNet+ALL2 are shown in Fig. 5. These results suggest that web images
reduced the ambiguity of queries’ semantics by providing hints on their visualconcepts. For example, with sentence (1) “A man is playing a keyboard,” retrieval results of GoogleNet+VS includes two videos of a keyboard on a laptop
as well as one on a musical instrument. On the other hand, all top-3 results by
GoogleNet+ALL2 are about musical instruments. Compared to GoogLeNet+VI,
our full model obtained more videos with relevant content. Moreover, the result

12

M. Otani et al.
Query Video

GoogLeNet+All2

GoogLeNet+VS

1. A man is cutting a paper.
2. A man is cutting a paper by hands.
3. Someone is cutting the carrot into small
pieces.

1. Someone is cutting the carrot into small
pieces.
2. A person cuts a sock with scissors.
3. An oriental lady is cutting a carrot into
thin pieces.

1. A woman is talking while applying
eyeshadow.
2. A woman applies Joker makeup to a
man's face.
3. A woman is applying cosmetics to a man.

1. A woman is singing.
2. A woman is singing.
3. A woman wearing a headset is singing
into a large microphone.

1. A pair of zebras are playing with eachother.
2. The zebras are playing.
3. A pair of zebras is nuzzling.

1. Leopards are congregating.
2. A group of deers are crossing road.
3. A pair of zebras is nuzzling.

1. A man is playing keyboards.
2. A boy is playing a grand piano.
3. A boy is playing guitar.

1. A little boy is playing piano.
2. A little boy is playing a grand piano.
3. A boy is playing a piano.

Fig. 6. Examples of top-3 retrieved sentences. Left: Query videos. Center: Top3 retrieved sentences by GoogLeNet+ALL2 . Right: Top-3 retrieved sentences by
GoogLeNet+VS.

of query (6) indicates that our model can recover from irrelevant image search
results by combining a query sentence.
Some examples of sentence retrieval results are shown in Fig. 6. While ourfull model may retrieve sentences that disagree with query videos in details, mostof the retrieved sentences are relevant to query videos.
4.4

Comparison to Prior Work

The approach for image and sentence retrieval by Kiros et al. [16] applies linear
transformations to CNN-based image and RNN-based sentence representations
to embed them into a common space. Note that their model was designed forthe image and sentence retrieval tasks; thus, we extracted the middle frame asa keyframe and trained the model with pairs of a keyframe and a sentence.
Xu et al. [34] introduced neural network-based embedding models for videos
and sentences. Their approach embeds videos and SVO triplets extracted fromsentences into an embedding space. Kiros et al.’s and Xu et al.’s approaches aredenoted by ST and DVCT, respectively.
Scores in Table 1 indicates that our model clearly outperformed prior workin both video and sentence retrieval tasks. There is a significant difference inperformance of DVCT and others. ST and ours encode all words in a sentence,
while DVCT only encodes its SVO triplets. This suggests that using all words
in a sentence together with an RNN is necessary to get good embeddings.

Learning Joint Representations of Videos and Sentences
“a”

“man”

“is”

“running”“<eos>”

13

softmax

LSTM

LSTM

LSTM

LSTM

LSTM

LSTM

“<bos>”

“a”

“man”

“is”

“running”

Fig. 7. Illustration of the decoder model. “<bos>” is a tag denoting the beginning ofa sentence, and “<eos>” is the end of a sentence.

Women are dancing.

A hamster eats seeds.

A man is riding a motorcycle.

A woman is riding ahorse.

A man is playing a piano.

A man is slicing a
potato.

Fig. 8. Sentences generated from our video embeddings. The sentence in red is a failure.
Table 3. Scores of generated sentences. TVNL+Extra Data is the TVNL model pretrained on the Flickr30k [36] and the COCO2014 [2] datasets.
ModelsTVNL [31]TVNL+Extra Data
DVETS [35]
Ours

5

CIDEr
51.67
41.62

BLEU
31.19
33.29
41.92
33.69

METEOR
26.87
29.07
29.60
28.47

Video Description Generation

Automatic description generation for images [32, 6] and videos [24, 31, 35] is another task to associate images or videos with sentences. As an application of ourmodels, we performed the description generation task using our video embeddings. To analyze the information encoded by our video embedding, we trained
a decoder that produces descriptions from our video embeddings. A basic approach for description generation is to use long-short term memory (LSTM) that

14

M. Otani et al.

produces a sequence of probabilities over a vocabulary conditioned on visual representations [32, 31]. We trained an LSTM as a decoder of video embeddings (Fig.
7). The decoder predicts the next word based on word vector wt at each time
step t as:[at it ft ot ]T = Wu wt + bu + Wl ht−1 ,(11)ct = tanh(at )σ(it ) + ct−1 σ(ft ),(12)ht = tanh(ct )σ(ot ),(13)

pt = softmax(Wp ht + bp )(14)

where Wu , Wl ∈ R4dw ×dw and bu ∈ R4dw are parameters of the LSTM, and[at it ft ot ]T is a column vector that is a concatenation of at , it , ft , ot ∈ Rdw .
The matrix Wp and the vector bp encode the hidden state into a vector withthe vocabulary size. The output pt is the probabilities over the vocabulary. We
built a vocabulary consisting of all words in the YouTube dataset and special
tags, i.e., begin-of-sentence (“<bos >”) and end-of-sentence (“<eos >”). The
generative process is terminated when “<eos >” is produced. We trained thedecoder using the YouTube dataset. We computed the video embedding φv (X)using GoogLeNet+ALL2 as an input to the LSTM at t = 0. We trained thedecoder by minimizing the cross entropy loss. During training, we fixed theparameters of our embedding models.
Figure 8 shows generated sentences. Although video embeddings were trained
for retrieval tasks and not finetuned for the decoder, we observed that mostgenerated sentences were semantically relevant to their original videos.
We evaluated generated sentences with the COCO description evaluation. We
found that the scores were comparable to prior work (Table 3). This indicatesthat our model efficiently encoded videos, maintaining their semantics. Moreover,this result suggests that our embeddings can be applied to other tasks that
require joint representations of videos and sentences.

6

Conclusion

We presented a video and sentence retrieval framework that incorporates web
images to bridge between sentences and videos. Specifically, we collected web
image search results in order to disambiguate semantics of a sentence. We developed neural network-based embedding models for video, sentence, and image
inputs which fuses sentence and image representations. We jointly trained video
and sentence embeddings using the YouTube dataset. Our experiments demonstrated the advantage of incorporating additional web images, and our approachclearly outperformed prior work in the both video and sentence retrieval tasks.
Furthermore, by decoding descriptions from video embeddings, we demonstrated
that rich semantics of videos were efficiently encoded in our video embeddings.
Our future work includes developing a video embedding that considers temporal structures of videos. It would be also interesting to investigate what kind ofsentences benefit from image search results, and how to collect efficient images.

Learning Joint Representations of Videos and Sentences

15

References
1. Chen, D.L., Dolan, W.B.: Collecting highly parallel data for paraphrase evaluation.
In: ACL. pp. 190–200 (2011)
2. Chen, X., Fang, H., Lin, T., Vedantam, R., Gupta, S., Dollr, P., Zitnick, C.L.:
Microsoft COCO Captions: Data collection and evaluation server. arXiv preprint
arXiv:1504.00325 p. 7 pages (2015)
3. Chopra, S., Hadsell, R., LeCun, Y.: Learning a similarity metric discriminatively,
with application to face verification. In: CVPR. pp. 539–546 (2005)
4. Datta, R., Joshi, D., Li, J., Wang, J.Z.: Image retrieval: Ideas, influences, andtrends of the new age. ACM Computing Surveys 40(2), 5:1–5:60 (2008)
5. Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., Darrell, T.:
DeCAF: A deep convolutional activation feature for generic visual recognition. In:
ICML. pp. 647–655 (2014)
6. Fang, H., Gupta, S., Iandola, F., Srivastava, R.K., Deng, L., Dollar, P., Gao, J.,
He, X., Mitchell, M., Platt, J.C., Zitnick, C.L., Zweig, G.: From captions to visualconcepts and back. In: CVPR. pp. 1473–1482 (2015)
7. Farhadi, A., Hejrati, M., Sadeghi, M.A., Young, P., Rashtchian, C., Hockenmaier,J., Forsyth, D.: Every picture tells a story: Generating sentences from images. In:
ECCV. pp. 15–29 (2010)
8. Fergus, R., Fei-Fei, L., Perona, P., Zisserman, A.: Learning object categories from
Google’s image search. In: ICCV. pp. 1816–1823 (2005)
9. Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Ranzato, M.A., Mikolov,
T.: DeViSE: A deep visual-semantic embedding model. In: NIPS, pp. 2121–2129(2013)
10. Girshick, R., Donahue, J., Darrell, T., Berkeley, U.C., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: CVPR. pp.
580–587 (2014)
11. Guadarrama, S., Venugopalan, S., Austin, U.T., Krishnamoorthy, N., Mooney, R.,
Malkarnenkar, G., Darrell, T., Berkeley, U.C.: YouTube2Text: Recognizing anddescribing arbitrary activities using semantic hierarchies and zero-shot recognition.
In: ICCV. pp. 2712–2719 (2013)
12. Gygli, M., Grabner, H., Van Gool, L.: Video summarization by learning submodular mixtures of objectives. In: CVPR. pp. 3090–3098 (2015)13. Johnson, J., Ballan, L., Fei-Fei, L.: Love thy neighbors: Image annotation by exploiting image metadata. In: ICCV. pp. 4624 – 4632 (2015)
14. Karpathy, A., Joulin, A., Fei-Fei, L.: Deep fragment embeddings for bidirectional
image sentence mapping. In: NIPS, pp. 1889–1897 (2014)
15. Kingma, D., Ba, J.: Adam: A method for stochastic optimization. In: ICLR. p. 11
pages (2015)
16. Kiros, R., Zhu, Y., Salakhutdinov, R.R., Zemel, R., Urtasun, R., Torralba, A.,
Fidler, S.: Skip-thought vectors. In: NIPS. pp. 3276–3284 (2015)
17. Le, Q.V., Mikolov, T.: Distributed representations of sentences and documents. In:
International Conference on Machine Learning (ICML). pp. 1188–1196 (2014)
18. Lin, D., Fidler, S., Kong, C., Urtasun, R.: Visual semantic search: Retrieving videos
via complex textual queries. In: CVPR. pp. 2657–2664 (2014)
19. Lin, T.Y., Belongie, S., Hays, J.: Learning deep representations for ground-to-aerial
geolocalization. In: CVPR. pp. 5007–5015 (2015)
20. Maybank, S.: A survey on visual content-based video indexing and retrieval. IEEE
Trans. Systems, Man, and Cybernetics, Part C (Applications and Reviews) 41(6),797–819 (2011)

16

M. Otani et al.

21. Ordonez, V., Kulkarni, G., Berg, T.: Im2Text: Describing images using 1 million
captioned photographs. In: NIPS. pp. 1143–1151 (2011)
22. Rashtchian, C., Young, P., Hodosh, M., Hockenmaier, J.: Collecting image annotations using amazon’s mechanical turk. In: NAACL-HLT. pp. 139–147 (2010)
23. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object
detection with region proposal networks. In: NIPS, pp. 91–99 (2015)
24. Rohrbach, M., Qiu, W., Titov, I., Thater, S., Pinkal, M., Schiele, B.: Translating
video content to natural language descriptions. In: ICCV. pp. 433–440 (2013)
25. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.,
Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: Imagenet large
scale visual recognition challenge. International Journal of Computer Vision 115(3),211–252 (2015)26. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale
image recoginition. In: ICLR. p. 14 pages (2015)27. Socher, R., Ganjoo, M., Manning, C.D., Ng, A.Y.: Zero-shot learning throughcross-modal transfer. In: NIPS. pp. 935–943 (2013)
28. Song, Y., Vallmitjana, J., Stent, A., Jaimes, A.: TVSum : Summarizing web videos
using titles. In: CVPR. pp. 5179–5187 (2015)29. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D.,
Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: CVPR. pp.
1–9 (2015)
30. Tokui, S., Oono, K., Hido, S., Clayton, J.: Chainer: A next-generation open source
framework for deep learning. In: NIPS. p. 6 pages (2015)31. Venugopalan, S., Xu, H., Donahue, J., Rohrbach, M., Mooney, R., Saenko, K.:
Translating videos to natural language using deep recurrent neural networks. In:
NAACL-HLT. pp. 1494–1504 (2014)32. Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image
caption generator. In: CVPR. pp. 3156–3164 (2015)
33. Wang, X., Gupta, A.: Unsupervised learning of visual representations using videos.
In: ICCV. pp. 2794–2802 (2015)
34. Xu, R., Xiong, C., Chen, W., Corso, J.: Jointly modeling deep video and compositional text to bridge vision and language in a unified framework. In: AAAI. pp.
2346–2352 (2015)35. Yao, L., Ballas, N., Larochelle, H., Courville, A.: Describing videos by exploiting
temporal structure. In: ICCV. pp. 4507 – 4515 (2015)36. Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual
denotations: New similarity metrics for semantic inference over event descriptions.
Transactions of the Association for Computational Linguistics 2, 67–78 (2014),
https://tacl2013.cs.columbia.edu/ojs/index.php/tacl/article/view/229
37. Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., Fidler,
S.: Aligning books and movies: Towards story-like visual explanations by watchingmovies and reading books. In: IEEE International Conference on Computer Vision(ICCV). pp. 19–27 (2015)