Deep Visual-Semantic Alignments for Generating Image Descriptions
Andrej Karpathy
Li Fei-Fei
Department of Computer Science, Stanford University{karpathy,feifeili}@cs.stanford.edu

Abstract
We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions tolearn about the inter-modal correspondences between language and visual data. Our alignment model is based on anovel combination of Convolutional Neural Networks overimage regions, bidirectional Recurrent Neural Networks
over sentences, and a structured objective that aligns thetwo modalities through a multimodal embedding. We thendescribe a Multimodal Recurrent Neural Network architecture that uses the inferred alignments to learn to generate
novel descriptions of image regions. We demonstrate that
our alignment model produces state of the art results in retrieval experiments on Flickr8K, Flickr30K and MSCOCO
datasets. We then show that the generated descriptions significantly outperform retrieval baselines on both full images
and on a new dataset of region-level annotations.

1. Introduction
A quick glance at an image is sufficient for a human topoint out and describe an immense amount of details about
the visual scene [14]. However, this remarkable ability has
proven to be an elusive task for our visual recognition models. The majority of previous work in visual recognition
has focused on labeling images with a fixed set of visual
categories and great progress has been achieved in these endeavors [45, 11]. However, while closed vocabularies of visual concepts constitute a convenient modeling assumption,
they are vastly restrictive when compared to the enormous
amount of rich descriptions that a human can compose.
Some pioneering approaches that address the challenge ofgenerating image descriptions have been developed [29,13]. However, these models often rely on hard-coded visualconcepts and sentence templates, which imposes limits on
their variety. Moreover, the focus of these works has beenon reducing complex visual scenes into a single sentence,
which we consider to be an unnecessary restriction.
In this work, we strive to take a step towards the goal of

Figure 1. Motivation/Concept Figure: Our model treats language
as a rich label space and generates descriptions of image regions.

generating dense descriptions of images (Figure 1). The
primary challenge towards this goal is in the design of amodel that is rich enough to simultaneously reason about
contents of images and their representation in the domainof natural language. Additionally, the model should be freeof assumptions about specific hard-coded templates, rules
or categories and instead rely on learning from the training
data. The second, practical challenge is that datasets of image captions are available in large quantities on the internet[21, 58, 37], but these descriptions multiplex mentions ofseveral entities whose locations in the images are unknown.
Our core insight is that we can leverage these large imagesentence datasets by treating the sentences as weak labels,
in which contiguous segments of words correspond to someparticular, but unknown location in the image. Our approach is to infer these alignments and use them to learn
a generative model of descriptions. Concretely, our contributions are twofold:• We develop a deep neural network model that infers the latent alignment between segments of sentences and the region of the image that they describe.

Our model associates the two modalities through acommon, multimodal embedding space and a structured objective. We validate the effectiveness of thisapproach on image-sentence retrieval experiments inwhich we surpass the state-of-the-art.
• We introduce a multimodal Recurrent Neural Networkarchitecture that takes an input image and generates
its description in text. Our experiments show that thegenerated sentences significantly outperform retrievalbased baselines, and produce sensible qualitative predictions. We then train the model on the inferred correspondences and evaluate its performance on a newdataset of region-level annotations.
We make code, data and annotations publicly available. 1

2. Related Work
Dense image annotations. Our work shares the high-levelgoal of densely annotating the contents of images withmany works before us. Barnard et al. [2] and Socher et
al. [48] studied the multimodal correspondence between
words and images to annotate segments of images. Several works [34, 18, 15, 33] studied the problem of holistic
scene understanding in which the scene type, objects andtheir spatial support in the image is inferred. However, thefocus of these works is on correctly labeling scenes, objects
and regions with a fixed set of categories, while our focus ison richer and higher-level descriptions of regions.
Generating descriptions. The task of describing images
with sentences has also been explored. A number of approaches pose the task as a retrieval problem, where themost compatible annotation in the training set is transferred
to a test image [21, 49, 13, 43, 23], or where training annotations are broken up and stitched together [30, 35, 31].
Several approaches generate image captions based on fixed
templates that are filled based on the content of the image[19, 29, 13, 55, 56, 9, 1] or generative grammars [42, 57],
but this approach limits the variety of possible outputs.
Most closely related to us, Kiros et al. [26] developed a logbilinear model that can generate full sentence descriptions
for images, but their model uses a fixed window context
while our Recurrent Neural Network (RNN) model conditions the probability distribution over the next word in a sentence on all previously generated words. Multiple closely
related preprints appeared on Arxiv during the submissionof this work, some of which also use RNNs to generate image descriptions [38, 54, 8, 25, 12, 5]. Our RNN is simpler
than most of these approaches but also suffers in performance. We quantify this comparison in our experiments.
Grounding natural language in images. A number of approaches have been developed for grounding text in the vi1 cs.stanford.edu/people/karpathy/deepimagesent

sual domain [27, 39, 60, 36]. Our approach is inspired by
Frome et al. [16] who associate words and images througha semantic embedding. More closely related is the work of
Karpathy et al. [24], who decompose images and sentences
into fragments and infer their inter-modal alignment using aranking objective. In contrast to their model which is basedon grounding dependency tree relations, our model aligns
contiguous segments of sentences which are more meaningful, interpretable, and not fixed in length.
Neural networks in visual and language domains. Multiple approaches have been developed for representing images and words in higher-level representations. On the image side, Convolutional Neural Networks (CNNs) [32, 28]
have recently emerged as a powerful class of models forimage classification and object detection [45]. On the sentence side, our work takes advantage of pretrained word
vectors [41, 22, 3] to obtain low-dimensional representations of words. Finally, Recurrent Neural Networks havebeen previously used in language modeling [40, 50], but weadditionally condition these models on images.

3. Our Model
Overview. The ultimate goal of our model is to generate
descriptions of image regions. During training, the input
to our model is a set of images and their corresponding
sentence descriptions (Figure 2). We first present a modelthat aligns sentence snippets to the visual regions that theydescribe through a multimodal embedding. We then treat
these correspondences as training data for a second, multimodal Recurrent Neural Network model that learns to generate the snippets.

3.1. Learning to align visual and language data
Our alignment model assumes an input dataset of images
and their sentence descriptions. Our key insight is that sentences written by people make frequent references to someparticular, but unknown location in the image. For example, in Figure 2, the words “Tabby cat is leaning” refer tothe cat, the words “wooden table” refer to the table, etc.
We would like to infer these latent correspondences, withthe eventual goal of later learning to generate these snippetsfrom image regions. We build on the approach of Karpathy
et al. [24], who learn to ground dependency tree relations
to image regions with a ranking objective. Our contribution is in the use of bidirectional recurrent neural network
to compute word representations in the sentence, dispensing of the need to compute dependency trees and allowing
unbounded interactions of words and their context in thesentence. We also substantially simplify their objective andshow that both modifications improve ranking performance.
We first describe neural networks that map words and image
regions into a common, multimodal embedding. Then we
introduce our novel objective, which learns the embedding

Figure 2. Overview of our approach. A dataset of images and their sentence descriptions is the input to our model (left). Our model firstinfers the correspondences (middle, Section 3.1) and then learns to generate novel descriptions (right, Section 3.2).

representations so that semantically similar concepts acrossthe two modalities occupy nearby regions of the space.
3.1.1

Representing images

Following prior work [29, 24], we observe that sentence descriptions make frequent references to objects and their attributes. Thus, we follow the method of Girshick et al. [17]to detect objects in every image with a Region Convolutional Neural Network (RCNN). The CNN is pre-trained on
ImageNet [6] and finetuned on the 200 classes of the ImageNet Detection Challenge [45]. Following Karpathy et al.
[24], we use the top 19 detected locations in addition to thewhole image and compute the representations based on the
pixels Ib inside each bounding box as follows:
v = Wm [CNNθc (Ib )] + bm ,(1)

where CNN(Ib ) transforms the pixels inside bounding box
Ib into 4096-dimensional activations of the fully connected
layer immediately before the classifier. The CNN parameters θc contain approximately 60 million parameters. The
matrix Wm has dimensions h × 4096, where h is the sizeof the multimodal embedding space (h ranges from 10001600 in our experiments). Every image is thus represented
as a set of h-dimensional vectors {vi | i = 1 . . . 20}.
3.1.2

Representing sentences

To establish the inter-modal relationships, we would liketo represent the words in the sentence in the same hdimensional embedding space that the image regions occupy. The simplest approach might be to project every individual word directly into this embedding. However, thisapproach does not consider any ordering and word context
information in the sentence. An extension to this idea is
to use word bigrams, or dependency tree relations as previously proposed [24]. However, this still imposes an arbitrary maximum size of the context window and requires
the use of Dependency Tree Parsers that might be trained on
unrelated text corpora.
To address these concerns, we propose to use a Bidirectional Recurrent Neural Network (BRNN) [46] to computethe word representations. The BRNN takes a sequence of

N words (encoded in a 1-of-k representation) and transforms each one into an h-dimensional vector. However, therepresentation of each word is enriched by a variably-sizedcontext around that word. Using the index t = 1 . . . N todenote the position of a word in a sentence, the precise formof the BRNN is as follows:
xt = Ww It(2)

et = f (We xt + be )(3)

hft = f (et + Wf hft−1 + bf )hbt = f (et + Wb hbt+1 + bb )(4)

st = f (Wd (hft + hbt ) + bd ).

(6)

(5)

Here, It is an indicator column vector that has a single one
at the index of the t-th word in a word vocabulary. The
weights Ww specify a word embedding matrix that we initialize with 300-dimensional word2vec [41] weights andkeep fixed due to overfitting concerns. However, in practice we find little change in final performance when these
vectors are trained, even from random initialization. Notethat the BRNN consists of two independent streams of processing, one moving left to right (hft ) and the other right toleft (hbt ) (see Figure 3 for diagram). The final h-dimensional
representation st for the t-th word is a function of both theword at that location and also its surrounding context in thesentence. Technically, every st is a function of all words inthe entire sentence, but our empirical finding is that the final
word representations (st ) align most strongly to the visualconcept of the word at that location (It ).
We learn the parameters We , Wf , Wb , Wd and the respective biases be , bf , bb , bd . A typical size of the hidden representation in our experiments ranges between 300-600 dimensions. We set the activation function f to the rectified
linear unit (ReLU), which computes f : x 7→ max(0, x).
3.1.3

Alignment objective

We have described the transformations that map every image and sentence into a set of vectors in a common hdimensional space. Since the supervision is at the level ofentire images and sentences, our strategy is to formulate animage-sentence score as a function of the individual regionword scores. Intuitively, a sentence-image pair should havea high matching score if its words have a confident support
in the image. The model of Karpathy et a. [24] interprets thedot product viT st between the i-th region and t-th word as ameasure of similarity and use it to define the score between
image k and sentence l as:XX
Skl =max(0, viT st ).
(7)t∈gl i∈gk

Here, gk is the set of image fragments in image k and gl
is the set of sentence fragments in sentence l. The indicesk, l range over the images and sentences in the training set.
Together with their additional Multiple Instance Learning
objective, this score carries the interpretation that a sentence
fragment aligns to a subset of the image regions whenever
the dot product is positive. We found that the following
reformulation simplifies the model and alleviates the needfor additional objectives and their hyperparameters:X
Skl =maxi∈gk viT st .
(8)
t∈gl

Here, every word st aligns to the single best image region.
As we show in the experiments, this simplified model also
leads to improvements in the final ranking performance.
Assuming that k = l denotes a corresponding image andsentence pair, the final max-margin, structured loss remains:

Figure 3. Diagram for evaluating the image-sentence score Skl .
Object regions are embedded with a CNN (left). Words (enrichedby their context) are embedded in the same multimodal space witha BRNN (right). Pairwise similarities are computed with innerproducts (magnitudes shown in grayscale) and finally reduced toimage-sentence score with Equation 8.

alignment to the same region. Concretely, given a sentencewith N words and an image with M bounding boxes, we
introduce the latent alignment variables aj ∈ {1 . . . M } forj = 1 . . . N and formulate an MRF in a chain structure
along the sentence as follows:
E(a) =XψjU (aj ) +j=1...N

C(θ) =Xh X
kmax(0, Skl − Skk + 1)

(9)

l

{z

|

rank images

}

i
X
+
max(0, Slk − Skk + 1) .
l

|

{z

rank sentences}

This objective encourages aligned image-sentences pairs tohave a higher score than misaligned pairs, by a margin.
3.1.4

Decoding text segment alignments to images

Consider an image from the training set and its corresponding sentence. We can interpret the quantity viT st as the unnormalized log probability of the t-th word describing anyof the bounding boxes in the image. However, since we areultimately interested in generating snippets of text insteadof single words, we would like to align extended, contiguous sequences of words to a single bounding box. Note thatthe naı̈ve solution that assigns each word independently tothe highest-scoring region is insufficient because it leads towords getting scattered inconsistently to different regions.
To address this issue, we treat the true alignments as latentvariables in a Markov Random Field (MRF) where the binary interactions between neighboring words encourage anX

ψjB (aj , aj+1 ) (10)j=1...N −1

ψjU (aj = t) = viT st(11)

ψjB (aj , aj+1 ) = β 1[aj = aj+1 ].

(12)Here, β is a hyperparameter that controls the affinity towards longer word phrases. This parameter allows us tointerpolate between single-word alignments (β = 0) andaligning the entire sentence to a single, maximally scoring
region when β is large. We minimize the energy to find thebest alignments a using dynamic programming. The outputof this process is a set of image regions annotated with segments of text. We now describe an approach for generatingnovel phrases based on these correspondences.

3.2. Multimodal Recurrent Neural Network forgenerating descriptions
In this section we assume an input set of images and their
textual descriptions. These could be full images and theirsentence descriptions, or regions and text snippets, as inferred in the previous section. The key challenge is in thedesign of a model that can predict a variable-sized sequenceof outputs given an image. In previously developed language models based on Recurrent Neural Networks (RNNs)[40, 50, 10], this is achieved by defining a probability distribution of the next word in a sequence given the current word
and context from previous time steps. We explore a simple

but effective extension that additionally conditions the generative process on the content of an input image. More formally, during training our Multimodal RNN takes the image
pixels I and a sequence of input vectors (x1 , . . . , xT ). It
then computes a sequence of hidden states (h1 , . . . , ht ) anda sequence of outputs (y1 , . . . , yt ) by iterating the following
recurrence relation for t = 1 to T :bv = Whi [CNNθc (I)]ht = f (Whx xt + Whh ht−1 + bh + 1(t = 1)
yt = sof tmax(Woh ht + bo ).

(13)bv ) (14)(15)

In the equations above, Whi , Whx , Whh , Woh , xi and bh , bo
are learnable parameters, and CNNθc (I) is the last layer ofa CNN. The output vector yt holds the (unnormalized) log
probabilities of words in the dictionary and one additional
dimension for a special END token. Note that we provide
the image context vector bv to the RNN only at the first
iteration, which we found to work better than at each time
step. In practice we also found that it can help to also pass
both bv , (Whx xt ) through the activation function. A typicalsize of the hidden layer of the RNN is 512 neurons.
RNN training. The RNN is trained to combine a word (xt ),the previous context (ht−1 ) to predict the next word (yt ).
We condition the RNN’s predictions on the image information (bv ) via bias interactions on the first step. The training
proceeds as follows (refer to Figure 4): We set h0 = ~0, x1 toa special START vector, and the desired label y1 as the firstword in the sequence. Analogously, we set x2 to the word
vector of the first word and expect the network to predict
the second word, etc. Finally, on the last step when xT represents the last word, the target label is set to a special END
token. The cost function is to maximize the log probability
assigned to the target labels (i.e. Softmax classifier).
RNN at test time. To predict a sentence, we compute theimage representation bv , set h0 = 0, x1 to the START vector and compute the distribution over the first word y1 . We
sample a word from the distribution (or pick the argmax),
set its embedding vector as x2 , and repeat this process until
the END token is generated. In practice we found that beam
search (e.g. beam size 7) can improve results.

3.3. Optimization
We use SGD with mini-batches of 100 image-sentence pairs
and momentum of 0.9 to optimize the alignment model. We
cross-validate the learning rate and the weight decay. We
also use dropout regularization in all layers except in therecurrent layers [59] and clip gradients elementwise at 5
(important). The generative RNN is more difficult to optimize, party due to the word frequency disparity between
rare words and common words (e.g. ”a” or the END token).
We achieved the best results using RMSprop [52], which isan adaptive step size method that scales the update of eachweight by a running average of its gradient norm.

Figure 4. Diagram of our multimodal Recurrent Neural Network
generative model. The RNN takes a word, the context from previous time steps and defines a distribution over the next word in thesentence. The RNN is conditioned on the image information at thefirst time step. START and END are special tokens.

4. Experiments
Datasets. We use the Flickr8K [21], Flickr30K [58] andMSCOCO [37] datasets in our experiments. These datasets
contain 8,000, 31,000 and 123,000 images respectively
and each is annotated with 5 sentences using Amazon
Mechanical Turk. For Flickr8K and Flickr30K, we use
1,000 images for validation, 1,000 for testing and the rest
for training (consistent with [21, 24]). For MSCOCO weuse 5,000 images for both validation and testing.
Data Preprocessing. We convert all sentences to lowercase, discard non-alphanumeric characters. We filter words
to those that occur at least 5 times in the training set,
which results in 2538, 7414, and 8791 words for Flickr8k,
Flickr30K, and MSCOCO datasets respectively.

4.1. Image-Sentence Alignment Evaluation
We first investigate the quality of the inferred text and imagealignments with ranking experiments. We consider a withheld set of images and sentences and retrieve items in one
modality given a query from the other by sorting based on
the image-sentence score Skl (Section 3.1.3). We report themedian rank of the closest ground truth result in the list and
Recall@K, which measures the fraction of times a correct
item was found among the top K results. The result of theseexperiments can be found in Table 1, and example retrievalsin Figure 5. We now highlight some of the takeaways.
Our full model outperforms previous work. First, ourfull model (“Our model: BRNN”) outperforms Socher et
al. [49] who trained with a similar loss but used a single
image representation and a Recursive Neural Network overthe sentence. A similar loss was adopted by Kiros et al.
[25], who use an LSTM [20] to encode sentences. We list
their performance with a CNN that is equivalent in power(AlexNet [28]) to the one used in this work, though similar to [54] they outperform our model with a more powerful
CNN (VGGNet [47], GoogLeNet [51]). “DeFrag” are theresults reported by Karpathy et al. [24]. Since we use different word vectors, dropout for regularization and different
cross-validation ranges and larger embedding sizes, we reimplemented their loss for a fair comparison (“Our imple-

ModelR@1

SDT-RNN (Socher et al. [49])Kiros et al. [25]Mao et al. [38]Donahue et al. [8]DeFrag (Karpathy et al. [24])
Our implementation of DeFrag [24]
Our model: DepTree edges
Our model: BRNN
Vinyals et al. [54] (more powerful CNN)

9.6
14.8
18.4
17.5
14.2
19.2
20.0
22.2
23

Our model: 1K test images
Our model: 5K test images

38.4
16.5

Image Annotation
R@5 R@10 Med r
Flickr30K
29.8
41.1
16
39.2
50.9
10
40.2
50.9
10
40.3
50.8
9
37.7
51.3
10
44.5
58.0
6.0
46.6
59.4
5.4
48.2
61.4
4.8
63
5
MSCOCO
69.9
80.5
1.0
39.2
52.0
9.0

R@1

Image Search
R@5 R@10

Med r

8.9
11.8
12.6
10.2
12.9
15.0
15.2
17

29.8
34.0
31.2
30.8
35.4
36.5
37.7
-

41.1
46.3
41.5
44.2
47.5
48.2
50.5
57

16
13
16
14
10.8
10.4
9.2
8

27.4
10.7

60.2
29.6

74.8
42.2

3.0
14.0

Table 1. Image-Sentence ranking experiment results. R@K is Recall@K (high is good). Med r is the median rank (low is good). In the
results for our models, we take the top 5 validation set models, evaluate each independently on the test set and then report the average
performance. The standard deviations on the recall values range from approximately 0.5 to 1.0.

Figure 5. Example alignments predicted by our model. For every test image above, we retrieve the most compatible test sentence andvisualize the highest-scoring region for each word (before MRF smoothing described in Section 3.1.4) and the associated scores (viT st ).
We hide the alignments of low-scoring words to reduce clutter. We assign each region an arbitrary color.

mentation of DeFrag”). Compared to other work that uses
AlexNets, our full model shows consistent improvement.
Our simpler cost function improves performance. We
strive to better understand the source of our performance.
First, we removed the BRNN and used dependency tree relations exactly as described in Karpathy et al. [24] (“Ourmodel: DepTree edges”). The only difference between thismodel and “Our reimplementation of DeFrag” is the new,simpler cost function introduced in Section 3.1.3. We seethat our formulation shows consistent improvements.
BRNN outperforms dependency tree relations. Furthermore, when we replace the dependency tree relations with
the BRNN we observe additional performance improvements. Since the dependency relations were shown to work
better than single words and bigrams [24], this suggests thatthe BRNN is taking advantage of contexts longer than two
words. Furthermore, our method does not rely on extracting
a Dependency Tree and instead uses the raw words directly.
MSCOCO results for future comparisons. We are notaware of other published ranking results on MSCOCO.

Therefore, we report results on a subset of 1,000 images
and the full set of 5,000 test images for future comparisons.
Note that the 5000 images numbers are lower since Recall@K is a function of test set size.
Qualitative. As can be seen from example groundings in
Figure 5, the model discovers interpretable visual-semantic
correspondences, even for small or relatively rare objects
such as an “accordion”. These would be likely missed bymodels that only reason about full images.
Learned region and word vector magnitudes. An appealing feature of our model is that it learns to modulate
the magnitude of the region and word embeddings. Dueto their inner product interaction, we observe that representations of visually discriminative words such as “kayaking, pumpkins“ have embedding vectors with higher magnitudes, which in turn translates to a higher influence on
the image-sentence score. Conversely, stop words such as“now, simply, actually, but” are mapped near the origin,which reduces their influence. See more analysis in supplementary material.

ModelNearest Neighbor
Mao et al. [38]
Google NIC [54]LRCN [8]MS Research [12]Chen and Zitnick [5]Our model

B-1
—
58
63
—
—
—
57.9

Flickr8K
B-2
B-3
—
—
28
23
41
27
—
—
—
—
—
—
38.3 24.5

B-4
—
—
—
—
—
14.1
16.0

B-1
—
55
66.3
58.8
—
—
57.3

Flickr30K
B-2
B-3
—
—
24
20
42.3 27.7
39.1 25.1————
36.9 24.0

B-4
—
—
18.3
16.5
—
12.6
15.7

B-1
48.0
—
66.6
62.8
—
—
62.5

B-2
28.1
—
46.1
44.2
—
—
45.0

MSCOCO 2014
B-3
B-4 METEOR
16.6 10.0
15.7
—
—
—
32.9 24.6
—
30.4
—
—
—
21.1
20.7
—
19.0
20.4
32.1 23.0
19.5

CIDEr
38.3
—
—
—
—
—
66.0

Table 2. Evaluation of full image predictions on 1,000 test images. B-n is BLEU score that uses up to n-grams. High is good in all columns.
For future comparisons, our METEOR/CIDEr Flickr8K scores are 16.7/31.8 and the Flickr30K scores are 15.3/24.7.

Figure 6. Example sentences generated by the multimodal RNN for test images. We provide many more examples on our project page.

4.2. Generated Descriptions: Fulframe evaluation
We now evaluate the ability of our RNN model to describe
images and regions. We first trained our Multimodal RNN
to generate sentences on full images with the goal of verifying that the model is rich enough to support the mappingfrom image data to sequences of words. For these full image experiments we use the more powerful VGGNet image
features [47]. We report the BLEU [44], METEOR [7] andCIDEr [53] scores computed with the coco-caption
code [4] 2 . Each method evaluates a candidate sentenceby measuring how well it matches a set of five reference
sentences written by humans.
Qualitative. The model generates sensible descriptions ofimages (see Figure 6), although we consider the last two
images failure cases. The first prediction “man in black
shirt is playing a guitar” does not appear in the training set.
However, there are 20 occurrences of “man in black shirt”
and 60 occurrences of “is paying guitar”, which the model
may have composed to describe the first image. In general,
we find that a relatively large portion of generated sentences(60% with beam size 7) can be found in the training data.
This fraction decreases with lower beam size; For instance,with beam size 1 this falls to 25%, but the performance also
deteriorates (e.g. from 0.66 to 0.61 CIDEr).
Multimodal RNN outperforms retrieval baseline. Our
first comparison is to a nearest neighbor retrieval baseline.
2 https://github.com/tylin/coco-caption

Here, we annotate each test image with a sentence of themost similar training set image as determined by L2 norm
over VGGNet [47] fc7 features. Table 2 shows that the Multimodal RNN confidently outperforms this retrieval method.
Hence, even with 113,000 train set images in MSCOCO
the retrieval approach is inadequate. Additionally, the RNN
takes only a fraction of a second to evaluate per image.
Comparison to other work. Several related models havebeen proposed in Arxiv preprints since the original submission of this work. We also include these in Table 2 for comparison. Most similar to our model is Vinyals et al. [54].
Unlike this work where the image information is communicated through a bias term on the first step, they incorporate it as a first word, they use a more powerful but more
complex sequence learner (LSTM [20]), a different CNN(GoogLeNet [51]), and report results of a model ensemble.
Donahue et al. [8] use a 2-layer factored LSTM (similarin structure to the RNN in Mao et al. [38]). Both models
appear to work worse than ours, but this is likely in largepart due to their use of the less powerful AlexNet [28] features. Compared to these approaches, our model prioritizes
simplicity and speed at a slight cost in performance.

4.3. Generated Descriptions: Region evaluation
We now train the Multimodal RNN on the correspondences
between image regions and snippets of text, as inferred by
the alignment model. To support the evaluation, we used
Amazon Mechanical Turk (AMT) to collect a new dataset

Figure 7. Example region predictions. We use our region-level multimodal RNN to generate text (shown on the right of each image) forsome of the bounding boxes in each image. The lines are grounded to centers of bounding boxes and the colors are chosen arbitrarily.

of region-level annotations that we only use at test time. The
labeling interface displayed a single image and asked annotators (we used nine per image) to draw five bounding boxes
and annotate each with text. In total, we collected 9,000 text
snippets for 200 images in our MSCOCO test split (i.e. 45
snippets per image). The snippets have an average length of
2.3 words. Example annotations include “sports car”, “elderly couple sitting”, “construction site”, “three dogs on
leashes”, “chocolate cake”. We noticed that asking annotators for grounded text snippets induces language statistics
different from those in full image captions. Our region annotations are more comprehensive and feature elements ofscenes that would rarely be considered salient enough to be
included in a single sentence sentence about the full image,such as “heating vent”, “belt buckle”, and “chimney”.
Qualitative. We show example region model predictionsin Figure 7. To reiterate the difficulty of the task, consider
for example the phrase “table with wine glasses” that is
generated on the image on the right in Figure 7. This phrase
only occurs in the training set 30 times. Each time it may
have a different appearance and each time it may occupy afew (or none) of our object bounding boxes. To generate
this string for the region, the model had to first correctly
learn to ground the string and then also learn to generate it.
Region model outperforms full frame model and ranking baseline. Similar to the full image description task, weevaluate this data as a prediction task from a 2D array of
pixels (one image region) to a sequence of words and record
the BLEU score. The ranking baseline retrieves training
sentence substrings most compatible with each region as
judged by the BRNN model. Table 3 shows that the region
RNN model produces descriptions most consistent with ourcollected data. Note that the fullframe model was trained
only on full images, so feeding it smaller image regions
deteriorates its performance. However, its sentences arealso longer than the region model sentences, which likely
negatively impacts the BLEU score. The sentence lengthis non-trivial to control for with an RNN, but we note thatthe region model also outperforms the fullframe model on
all other metrics: CIDEr 61.6/20.3, METEOR 15.8/13.3,
ROUGE 35.1/21.0 for region/fullframe respectively.

Model
Human agreement
Nearest Neighbor
RNN: Fullframe model
RNN: Region level model

B-1
61.5
22.9
14.2
35.2

B-2
45.2
10.5
6.0
23.0

B-3
30.1
0.0
2.2
16.1

B-4
22.0
0.0
0.0
14.8

Table 3. BLEU score evaluation of image region annotations.

4.4. Limitations
Although our results are encouraging, the Multimodal RNN
model is subject to multiple limitations. First, the model canonly generate a description of one input array of pixels at a
fixed resolution. A more sensible approach might be to use
multiple saccades around the image to identify all entities,
their mutual interactions and wider context before generating a description. Additionally, the RNN receives the image
information only through additive bias interactions, which
are known to be less expressive than more complicated multiplicative interactions [50, 20]. Lastly, our approach consists of two separate models. Going directly from an imagesentence dataset to region-level annotations as part of a single model trained end-to-end remains an open problem.

5. Conclusions
We introduced a model that generates natural language descriptions of image regions based on weak labels in form ofa dataset of images and sentences, and with very few hardcoded assumptions. Our approach features a novel ranking
model that aligned parts of visual and language modalities
through a common, multimodal embedding. We showedthat this model provides state of the art performance on
image-sentence ranking experiments. Second, we describeda Multimodal Recurrent Neural Network architecture thatgenerates descriptions of visual data. We evaluated its performance on both fullframe and region-level experiments
and showed that in both cases the Multimodal RNN outperforms retrieval baselines.
Acknowledgements.
We thank Justin Johnson and Jon Krause for helpful comments and discussions. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the GPUs
used for this research. This research is partially supported
by an ONR MURI grant, and NSF ISS-1115313.

References[1] A. Barbu, A. Bridge, Z. Burchill, D. Coroian, S. Dickinson, S. Fidler, A. Michaux, S. Mussman, S. Narayanaswamy,
D. Salvi, et al. Video in sentences out. arXiv preprint
arXiv:1204.2742, 2012. 2[2] K. Barnard, P. Duygulu, D. Forsyth, N. De Freitas, D. M.
Blei, and M. I. Jordan. Matching words and pictures. JMLR,
2003. 2[3] Y. Bengio, H. Schwenk, J.-S. Senécal, F. Morin, and J.-L.
Gauvain. Neural probabilistic language models. In Innovations in Machine Learning. Springer, 2006. 2[4] X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollar, and C. L. Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325,
2015. 7
[5] X. Chen and C. L. Zitnick. Learning a recurrent visual representation for image caption generation. CoRR,
abs/1411.5654, 2014. 2, 7
[6] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. FeiFei. Imagenet: A large-scale hierarchical image database. In
CVPR, 2009. 3[7] M. Denkowski and A. Lavie. Meteor universal: Language
specific translation evaluation for any target language. In
Proceedings of the EACL 2014 Workshop on Statistical Machine Translation, 2014. 7[8] J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach,
S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. arXiv preprint arXiv:1411.4389, 2014. 2, 6, 7
[9] D. Elliott and F. Keller. Image description using visual dependency representations. In EMNLP, pages 1292–1302,
2013. 2[10] J. L. Elman. Finding structure in time. Cognitive science,14(2):179–211, 1990. 4[11] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, andA. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303–
338, June 2010. 1[12] H. Fang, S. Gupta, F. Iandola, R. Srivastava, L. Deng,
P. Dollár, J. Gao, X. He, M. Mitchell, J. Platt, et al.
From captions to visual concepts and back. arXiv preprint
arXiv:1411.4952, 2014. 2, 7
[13] A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young,
C. Rashtchian, J. Hockenmaier, and D. Forsyth. Every picture tells a story: Generating sentences from images. In
ECCV. 2010. 1, 2
[14] L. Fei-Fei, A. Iyer, C. Koch, and P. Perona. What do weperceive in a glance of a real-world scene? Journal of vision,
7(1):10, 2007. 1[15] S. Fidler, A. Sharma, and R. Urtasun. A sentence is worth athousand pixels. In CVPR, 2013. 2[16] A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean,
T. Mikolov, et al. Devise: A deep visual-semantic embedding model. In NIPS, 2013. 2[17] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic
segmentation. In CVPR, 2014. 3[18] S. Gould, R. Fulton, and D. Koller. Decomposing a scene
into geometric and semantically consistent regions. In Computer Vision, 2009 IEEE 12th International Conference on,
pages 1–8. IEEE, 2009. 2[19] A. Gupta and P. Mannem. From image annotation to image description. In Neural information processing. Springer,
2012. 2[20] S. Hochreiter and J. Schmidhuber. Long short-term memory.
Neural computation, 9(8):1735–1780, 1997. 5, 7, 8
[21] M. Hodosh, P. Young, and J. Hockenmaier. Framing image
description as a ranking task: data, models and evaluation
metrics. Journal of Artificial Intelligence Research, 2013. 1,2, 5[22] R. JeffreyPennington and C. Manning. Glove: Global vectors for word representation. 2014. 2[23] Y. Jia, M. Salzmann, and T. Darrell. Learning cross-modality
similarity for multinomial data. In ICCV, 2011. 2[24] A. Karpathy, A. Joulin, and L. Fei-Fei. Deep fragment embeddings for bidirectional image sentence mapping. arXiv
preprint arXiv:1406.5679, 2014. 2, 3, 4, 5, 6
[25] R. Kiros, R. Salakhutdinov, and R. S. Zemel. Unifying
visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539, 2014. 2,5, 6[26] R. Kiros, R. S. Zemel, and R. Salakhutdinov. Multimodal
neural language models. ICML, 2014. 2[27] C. Kong, D. Lin, M. Bansal, R. Urtasun, and S. Fidler. What
are you talking about? text-to-image coreference. In CVPR,
2014. 2[28] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classification with deep convolutional neural networks. In
NIPS, 2012. 2, 5, 7[29] G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. C. Berg,and T. L. Berg. Baby talk: Understanding and generating
simple image descriptions. In CVPR, 2011. 1, 2, 3[30] P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, and
Y. Choi. Collective generation of natural image descriptions.
In ACL, 2012. 2[31] P. Kuznetsova, V. Ordonez, T. L. Berg, U. C. Hill, and
Y. Choi. Treetalk: Composition and compression of trees
for image descriptions. Transactions of the Association for
Computational Linguistics, 2(10):351–362, 2014. 2[32] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradientbased learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. 2[33] L.-J. Li and L. Fei-Fei. What, where and who? classifying
events by scene and object recognition. In ICCV, 2007. 2[34] L.-J. Li, R. Socher, and L. Fei-Fei. Towards total scene understanding: Classification, annotation and segmentation in
an automatic framework. In Computer Vision and Pattern
Recognition, 2009. CVPR 2009. IEEE Conference on, pages
2036–2043. IEEE, 2009. 2[35] S. Li, G. Kulkarni, T. L. Berg, A. C. Berg, and Y. Choi. Composing simple image descriptions using web-scale n-grams.
In CoNLL, 2011. 2[36] D. Lin, S. Fidler, C. Kong, and R. Urtasun. Visual semantic
search: Retrieving videos via complex textual queries. 2014.
2

[37] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. arXiv preprint arXiv:1405.0312,
2014. 1, 5
[38] J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille. Explain
images with multimodal recurrent neural networks. arXiv
preprint arXiv:1410.1090, 2014. 2, 6, 7
[39] C. Matuszek*, N. FitzGerald*, L. Zettlemoyer, L. Bo, andD. Fox. A Joint Model of Language and Perception for
Grounded Attribute Learning. In Proc. of the 2012 International Conference on Machine Learning, Edinburgh, Scotland, June 2012. 2[40] T. Mikolov, M. Karafiát, L. Burget, J. Cernockỳ, and S. Khudanpur. Recurrent neural network based language model. In
INTERSPEECH, 2010. 2, 4[41] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, andJ. Dean. Distributed representations of words and phrases
and their compositionality. In NIPS, 2013. 2, 3[42] M. Mitchell, X. Han, J. Dodge, A. Mensch, A. Goyal,
A. Berg, K. Yamaguchi, T. Berg, K. Stratos, and H. Daumé,III. Midge: Generating image descriptions from computer
vision detections. In EACL, 2012. 2[43] V. Ordonez, G. Kulkarni, and T. L. Berg. Im2text: Describing images using 1 million captioned photographs. In NIPS,
2011. 2[44] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: amethod for automatic evaluation of machine translation. In
Proceedings of the 40th annual meeting on association forcomputational linguistics, pages 311–318. Association for
Computational Linguistics, 2002. 7[45] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
A. C. Berg, and L. Fei-Fei. Imagenet large scale visual recognition challenge, 2014. 1, 2, 3
[46] M. Schuster and K. K. Paliwal. Bidirectional recurrent neuralnetworks. Signal Processing, IEEE Transactions on, 1997.
3[47] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. arXiv preprint
arXiv:1409.1556, 2014. 5, 7
[48] R. Socher and L. Fei-Fei. Connecting modalities: Semisupervised segmentation and annotation of images using unaligned text corpora. In CVPR, 2010. 2[49] R. Socher, A. Karpathy, Q. V. Le, C. D. Manning, and A. Y.
Ng. Grounded compositional semantics for finding and describing images with sentences. TACL, 2014. 2, 5, 6[50] I. Sutskever, J. Martens, and G. E. Hinton. Generating text
with recurrent neural networks. In ICML, 2011. 2, 4, 8
[51] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed,
D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv preprint
arXiv:1409.4842, 2014. 5, 7[52] T. Tieleman and G. E. Hinton. Lecture 6.5-rmsprop: Divide
the gradient by a running average of its recent magnitude.,2012. 5[53] R. Vedantam, C. L. Zitnick, and D. Parikh.
Cider:
Consensus-based image description evaluation.
CoRR,
abs/1411.5726, 2014. 7[54] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Showand tell: A neural image caption generator. arXiv preprint
arXiv:1411.4555, 2014. 2, 5, 6, 7[55] Y. Yang, C. L. Teo, H. Daumé III, and Y. Aloimonos.
Corpus-guided sentence generation of natural images. In
EMNLP, 2011. 2[56] B. Z. Yao, X. Yang, L. Lin, M. W. Lee, and S.-C. Zhu. I2t:
Image parsing to text description. Proceedings of the IEEE,
98(8):1485–1508, 2010. 2[57] M. Yatskar, L. Vanderwende, and L. Zettlemoyer. See noevil, say no evil: Description generation from densely labeled images. Lexical and Computational Semantics, 2014.
2[58] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL,
2014. 1, 5[59] W. Zaremba, I. Sutskever, and O. Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329,
2014. 5[60] C. L. Zitnick, D. Parikh, and L. Vanderwende. Learning thevisual interpretation of sentences. ICCV, 2013. 2