Title: BOOSTING IMAGE CAPTIONING WITH ATTRIBUTES

Abstract: Automatically describing an image with a natural language has been an emerging challenge in both fields of computer vision and natural language processing. In this paper, we present Long Short-Term Memory with Attributes (LSTM-A) -a novel architecture that integrates attributes into the successful Convolutional Neural Networks (CNNs) plus Recurrent Neural Networks (RNNs) image captioning framework, by training them in an end-to-end manner. To incorporate attributes, we construct variants of architectures by feeding image representations and attributes into RNNs in different ways to explore the mutual but also fuzzy relationship between them. Extensive experiments are conducted on COCO image captioning dataset and our framework achieves superior results when compared to state-of-the-art deep models. Most remarkably, we obtain METEOR/CIDEr-D of 25.2%/98.6% on testing data of widely used and publicly available splits in (Karpathy & Fei-Fei, 2015) when extracting image representations by GoogleNet and achieve to date top-1 performance on COCO captioning Leaderboard.

1. INTRODUCTION:

Accelerated by tremendous increase in Internet bandwidth and proliferation of sensor-rich mobile devices, image data has been generated, published and spread explosively, becoming an indispensable part of today's big data. This has encouraged the development of advanced techniques for a broad range of image understanding applications. A fundamental issue that underlies the success of these technological advances is the recognition (Szegedy et al., 2015; Simonyan & Zisserman, 2015; He et al., 2016) . Recently, researchers have strived to automatically describe the content of an image with a complete and natural sentence, which has a great potential impact for instance on robotic vision or helping visually impaired people. Nevertheless, this problem is very challenging, as description generation model should capture not only the objects or scenes presented in the image, but also be capable of expressing how the objects/scenes relate to each other in a nature sentence.

The main inspiration of recent attempts on this problem (Donahue et al., 2015; Vinyals et al., 2015; Xu et al., 2015; You et al., 2016) are from the advances by using RNNs in machine translation (Sutskever et al., 2014) , which is to translate a text from one language (e.g., English) to another (e.g., Chinese). The basic idea is to perform a sequence to sequence learning for translation, where an encoder RNN reads the input sequential sentence, one word at a time till the end of the sentence and then a decoder RNN is exploited to generate the sentence in target language, one word at each time step. Following this philosophy, it is natural to employ a CNN instead of the encoder RNN for image captioning, which is regarded as an image encoder to produce image representations.

While encouraging performances are reported, these CNN plus RNN image captioning methods translate directly from image representations to language, without explicitly taking more high-level semantic information from images into account. Furthermore, attributes are properties observed in images with rich semantic cues and have been proved to be effective in visual recognition (Parikh & Grauman, 2011) . A valid question is how to incorporate high-level image attributes into CNN plus RNN image captioning architecture as complementary knowledge in addition to image representations. We investigate particularly in this paper the architectures by exploiting the mutual relationship between image representations and attributes for enhancing image description generation. Specifically, to better demonstrate the impact of simultaneously utilizing the two kinds of representations, we devise variants of architectures by feeding them into RNN in different placements and moments, e.g., leveraging only attributes, inserting image representations first and then attributes or vice versa, and inputting image representations/attributes once or at each time step.

The main contribution of this work is the proposal of attribute augmented architectures by integrating the attributes into CNN plus RNN image captioning framework, which is a problem not yet fully understood in the literature. By leveraging more knowledge for building richer representations and description models, our work takes a further step forward to enhance image captioning and could have a direct impact of indicating a new direction of vision and language research. More importantly, the utilization of attributes also has a great potential to be an elegant solution of generating openvocabulary sentences, making image captioning system really practical.

2. RELATED WORK:

The research on image captioning has proceeded along three different dimensions: template-based methods (Kulkarni et al., 2013; Yang et al., 2011; Mitchell et al., 2012) , search-based approaches (Farhadi et al., 2010; Ordonez et al., 2011; Devlin et al., 2015) , and language-based models (Donahue et al., 2015; Kiros et al., 2014; Mao et al., 2014; Vinyals et al., 2015; Xu et al., 2015; Wu et al., 2016; You et al., 2016) .

The first direction, template-based methods, predefine the template for sentence generation which follows some specific rules of language grammar and split sentence into several parts (e.g., subject, verb, and object). With such sentence fragments, many works align each part with image content and then generate the sentence for the image. Obviously, most of them highly depend on the templates of sentence and always generate sentence with syntactical structure. For example, Kulkarni et al. employ Conditional Random Field (CRF) model to predict labeling based on the detected objects, attributes, and prepositions, and then generate sentence with a template by filling in slots with the most likely labeling (Kulkarni et al., 2013) . Similar in spirit, Yang et al. utilize Hidden Markov Model (HMM) to select the best objects, scenes, verbs, and prepositions with the highest log-likelihood ratio for template-based sentence generation in (Yang et al., 2011) . Furthermore, the traditional simple template is extended to syntactic trees in (Mitchell et al., 2012) which also starts from detecting attributes from image as description anchors and then connecting ordered objects with a syntactically well-formed tree, followed by adding necessary descriptive information.

Search-based approaches "generate" sentence for an image by selecting the most semantically similar sentences from sentence pool or directly copying sentences from other visually similar images. This direction indeed can achieve human-level descriptions as all sentences are from existing humangenerated sentences. The need to collect human-generated sentences, however, makes the sentence pool hard to be scaled up. Moreover, the approaches in this dimension cannot generate novel descriptions. For instance, in (Farhadi et al., 2010) , an intermediate meaning space based on the triplet of object, action, and scene is proposed to measure the similarity between image and sentence, where the top sentences are regarded as the generated sentences for the target image. Ordonez et al. (Ordonez et al., 2011) search images in a large captioned photo collection by using the combination of object, stuff, people, and scene information and transfer the associated sentences to the query image. Recently, a simple k-nearest neighbor retrieval model is utilized in (Devlin et al., 2015) and the best or consensus caption is selected from the returned candidate captions, which even performs as well as several state-of-the-art language-based models.

Different from template-based and search-based models, language-based models aim to learn the probability distribution in the common space of visual content and textual sentence to generate novel sentences with more flexible syntactical structures. In this direction, recent works explore such probability distribution mainly using neural networks for image captioning. For instance, in (Vinyals et al., 2015) , Vinyals et al. propose an end-to-end neural networks architecture by utilizing LSTM to generate sentence for an image, which is further incorporated with attention mechanism in (Xu et al., 2015) to automatically focus on salient objects when generating corresponding words. More recently, in (Wu et al., 2016) , high-level concepts/attributes are shown to obtain clear improvements on image captioning when injected into existing state-of-the-art RNN-based model and such visual attributes are further utilized as semantic attention in (You et al., 2016) to enhance image captioning.

In short, our work in this paper belongs to the language-based models. Different from most of the aforementioned language-based models which mainly focus on sentence generation by solely depending on image representations (Donahue et al., 2015; Kiros et al., 2014; Mao et al., 2014; Vinyals et al., 2015; Xu et al., 2015) or high-level attributes (Wu et al., 2016) , our work contributes by studying not only jointly exploiting image representations and attributes for image captioning, but also how the architecture can be better devised by exploring mutual relationship in between. It is also worth noting that (You et al., 2016) also additionally involve attributes for image captioning. Ours is fundamentally different in the way that (You et al., 2016) is as a result of utilizing attributes to model semantic attention to the locally previous words, as opposed to holistically employing attributes as a kind of complementary representations in this work.

3. BOOSTING IMAGE CAPTIONING WITH ATTRIBUTES:

In this paper, we devise our CNN plus RNN architectures to generate descriptions for images under the umbrella of additionally incorporating the detected high-level attributes. Specifically, we begin this section by presenting the problem formulation and followed by five variants of our image captioning frameworks with attributes.

3.1. PROBLEM FORMULATION:

Suppose we have an image I to be described by a textual sentence S, where S = {w 1 , w 2 , ..., w Ns } consisting of N s words. Let I ∈ R Dv and w t ∈ R Ds denote the D v -dimensional image representations of the image I and the D s -dimensional textual features of the t-th word in sentence S, respectively. Furthermore, we have feature vector A ∈ R Da to represent the probability distribution over the high-level attributes for image I. Specifically, we train the attribute detectors by using the weakly-supervised approach of Multiple Instance Learning (MIL) in (Fang et al., 2015) on image captioning benchmarks. For an attribute w a , one image I is regarded as a positive bag of regions (instances) if w a exists in image I's ground-truth sentences, and negative bag otherwise. By inputting all the bags into a noisy-OR MIL model, the probability of the bag b I which contains attribute w a is measured on the probabilities of all the regions in the bag as

EQUATION

where p wa i is the probability of the attribute w a predicted by region r i and can be calculated through a sigmoid layer after the last convolutional layer in the fully convolutional network. In particular, the dimension of convolutional activations from the last convolutional layer is x×x×h and h represents the representation dimension of each region, resulting in x × x response map which preserves the spatial dependency of the image. Then, a cross entropy loss is calculated based on the probabilities of all the attributes at the top of the whole architecture to optimize MIL model. With the learnt MIL model on image captioning dataset, we treat the final image-level response probabilities of all the attributes as A.

Inspired by the recent successes of probabilistic sequence models leveraged in statistical machine translation (Bahdanau et al., 2015; Sutskever et al., 2014) , we aim to formulate our image captioning models in an end-to-end fashion based on RNNs which encode the given image and/or its detected attributes into a fixed dimensional vector and then decode it to the target output sentence. Hence, the sentence generation problem we explore here can be formulated by minimizing the following energy loss function as

EQUATION

which is the negative log probability of the correct textual sentence given the image representations and detected attributes.

Since the model produces one word in the sentence at each time step, it is natural to apply chain rule to model the joint probability over the sequential words. Thus, the log probability of the sentence is given by the sum of the log probabilities over the word and can be expressed as

log Pr (S|I, A) = Ns t=1

log Pr ( wt| I, A, w0, . . . , wt-1).

(3)

By minimizing this loss, the contextual relationship among the words in the sentence can be guaranteed given the image and its detected attributes. We formulate this task as a variable-length sequence to sequence problem and model the parametric distribution Pr ( w t | I, A, w 0 , . . . , w t-1 ) in Eq.( 3) with Long Short-Term Memory (LSTM), which is a widely used type of RNN. The vector formulas for a LSTM layer forward pass are summarized as below. For time step t, x t and h t are the input and output vector respectively, T are input weights matrices, R are recurrent weight matrices and b are bias vectors. Sigmoid σ and hyperbolic tangent φ are element-wise non-linear activation functions. The dot product of two vectors is denoted with . Given inputs x t , h t-1 and c t-1 , the LSTM unit updates for time step t are:

g t = φ(T g x t + R g h t-1 + b g ), i t = σ(T i x t + R i h t-1 + b i ), f t = σ(T f x t + R f h t-1 + b f ), c t = g t i t + c t-1 f t , o t = σ(T o x t + R o h t-1 + b o ), h t = φ(c t ) o t ,

where g t , i t , f t , c t , o t , and h t are cell input, input gate, forget gate, cell state, output gate, and cell output of the LSTM, respectively.

3.2. LONG SHORT-TERM MEMORY WITH ATTRIBUTES:

Unlike the existing image captioning models in (Donahue et al., 2015; Vinyals et al., 2015) which solely encode image representations for sentence generation, our proposed Long Short-Term Memory with Attributes (LSTM-A) model additionally integrates the detected high-level attributes into LSTM. We devise five variants of LSTM-A for involvement of two design purposes. The first purpose is about where to feed attributes into LSTM and three architectures, i.e., LSTM-A 1 (leveraging only attributes), LSTM-A 2 (inserting image representations first) and LSTM-A 3 (feeding attributes first), are derived from this view. The second is about when to input attributes or image representations into LSTM and we design LSTM-A 4 (inputting image representations at each time step) and LSTM-A 5 (inputting attributes at each time step) for this purpose. An overview of LSTM-A architectures is depicted in Figure 1 .

3.2.1. LSTM-A 1 (LEVERAGING ONLY ATTRIBUTES):

Given the detected attributes, one natural way is to directly inject the attributes as representations at the initial time to inform the LSTM about the high-level attributes. This kind of architecture with only attributes input is named as LSTM-A 1 . It is also worth noting that the attributes-based model in (Wu et al., 2016) is similar to LSTM-A 1 and can be regarded as one special case of our LSTM-A.

Given the attribute representations A and the corresponding sentence W ≡ [w 0 , w 1 , ..., w Ns ], the LSTM updating procedure in LSTM-A 1 is as

x -1 = T a A, x t = T s w t , t ∈ {0, . . . , N s -1} and h t = f x t , t ∈ {0, . . . , N s -1} ,

where D e is the dimensionality of LSTM input, T a ∈ R De×Da and T s ∈ R De×Ds is the transformation matrix for attribute representations and textual features of word, respectively, and f is the updating function within LSTM unit. Please note that for the input sentence W ≡ [w 0 , . . . , w Ns ], we take w 0 as the start sign word to inform the beginning of sentence and w Ns as the end sign word which indicates the end of sentence. Both of the special sign words are included in our vocabulary. Most specifically, at the initial time step, the attribute representations are transformed as the input to LSTM, and then in the next steps, word embedding x t will be input into the L-STM along with the previous step's hidden state h t-1 . In each time step (except the initial step), we use the LSTM cell output h t to predict the next word. Here a softmax layer is applied after the LSTM layer to produce a probability distribution over all the D s words in the vocabulary as

Pr t+1 (w t+1 ) = exp T (w t+1 ) h h t w∈W exp T (w) h h t ,

where W is the word vocabulary space and T (w) h

is the parameter matrix in softmax layer.

3.2.2. LSTM-A 2 (INSERTING IMAGE REPRESENTATIONS FIRST):

To further leverage both image representations and high-level attributes in the encoding stage of our LSTM-A, we design the second architecture LSTM-A 2 by treating both of them as atoms in the input sequence to LSTM. Specifically, at the initial step, the image representations I are firstly transformed into LSTM to inform the LSTM about the image content, followed by the attribute representations A which are encoded into LSTM at the next time step to inform the high-level attributes. Then, LSTM decodes each output word based on previous word x t and previous step's hidden state h t-1 , which is similar to the decoding stage in LSTM-A 1 . The LSTM updating procedure in LSTM-A 2 is designed as

x -2 = T v I and x -1 = T a A, x t = T s w t , t ∈ {0, . . . , N s -1} and h t = f x t , t ∈ {0, . . . , N s -1} ,

where T v ∈ R De×Dv is the transformation matrix for image representations.

3.2.3. LSTM-A 3 (FEEDING ATTRIBUTES FIRST):

The third design LSTM-A 3 is similar to LSTM-A 2 as both designs utilize image representations and high-level attributes to form the input sequence to LSTM in the encoding stage, except that the orders of encoding are different. In LSTM-A 3 , the attribute representations are firstly encoded into LSTM and then the image representations are transformed into LSTM at the second time step. The whole LSTM updating procedure in LSTM-A 3 is as

x -2 = T a A and x -1 = T v I, x t = T s w t , t ∈ {0, . . . , N s -1} and h t = f x t , t ∈ {0, . . . , N s -1} .

3.2.4. LSTM-A 4 (INPUTTING IMAGE REPRESENTATIONS AT EACH TIME STEP):

Different from the former three designed architectures which mainly inject high-level attributes and image representations at the encoding stage of LSTM, we next modify the decoding stage in our LSTM-A by additionally incorporating image representations or high-level attributes. More specifically, in LSTM-A 4 , the attribute representations are injected once at the initial step to inform the LSTM about the high-level attributes, and then image representations are fed at each time step as an extra input to LSTM to emphasize the image content frequently among memory cells in LSTM. Hence, the LSTM updating procedure in LSTM-A 4 is:

x

-1 = T a A, x t = T s w t + T v I, t ∈ {0, . . . , N s -1} and h t = f x t , t ∈ {0, . . . , N s -1} .

3.2.5. LSTM-A 5 (INPUTTING ATTRIBUTES AT EACH TIME STEP):

The last design LSTM-A 5 is similar to LSTM-A 4 except that it firstly encodes image representations and then feeds attribute representations as an additional input to LSTM at each step in decoding stage to emphasize the high-level attributes frequently. Accordingly, the LSTM updating procedure in LSTM-A 5 is as

x -1 = T v I, x t = T s w t + T a A, t ∈ {0, . . . , N s -1} and h t = f x t , t ∈ {0, . . . , N s -1} .

4. EXPERIMENTS:

We conducted our experiments on COCO captioning dataset (COCO) (Lin et al., 2014) and evaluated our approaches for image captioning.

4.1. DATASET:

The dataset, COCO, is the most popular benchmark for image captioning, which contains 82,783 training images and 40,504 validation images. There are 5 human-annotated descriptions per image. As the annotations of the official testing set are not publicly available, we follow the widely used settings in prior works (You et al., 2016; Zhou et al., 2016) and take 82,783 images for training, 5,000 for validation and 5,000 for testing.

4.2. EXPERIMENTAL SETTINGS:

Data Preprocessing. Following (Karpathy & Fei-Fei, 2015) , we convert all the descriptions in training set to lower case and discard rare words which occur less than 5 times, resulting in the final vocabulary with 8,791 unique words in COCO dataset.

Features and Parameter Settings. Each word in the sentence is represented as "one-hot" vector (binary index vector in a vocabulary). For image representations, we take the output of 1,024-way pool5/7 × 7 s1 layer from GoogleNet (Szegedy et al., 2015) pre-trained on Imagenet ILSVRC12 dataset (Russakovsky et al., 2015) . For attribute representations, we select 1,000 most common words on COCO as the high-level attributes and train the attribute detectors with MIL model (Fang et al., 2015) purely on the training data of COCO, resulting in the final 1,000-way vector of probabilities of attributes. The dimension of the input and hidden layers in LSTM are both set to 1,024.

Implementation Details. We mainly implement our image captioning models based on Caffe (Jia et al., 2014) , which is one of widely adopted deep learning frameworks. Specifically, with an initial learning rate 0.01 and mini-batch size set 1,024, the objective value can decrease to 25% of the initial loss and reach a reasonable result after 50,000 iterations (about 123 epochs).

Testing Strategies. For sentence generation in testing stage, there are two common strategies. One is to choose the word with maximum probability at each time step and set it as LSTM input for next time step until the end sign word is emitted or the maximum length of sentence is reached. The other strategy is beam search which selects the top-k best sentences at each time step and considers them as the candidates to generate new top-k best sentences at the next time step. We adopt the second strategy and the beam size k is empirically set to 3.

Moreover, to avoid model-level overfitting, we utilize ensembling strategy to fuse the prediction results of 5 identical models as previous works (Vinyals et al., 2015; You et al., 2016) . Please note that all the 5 identical models are trained with different initializations separately.

Evaluation Metrics. For the evaluation of our proposed models, we adopt five metrics: BLEU@N (Papineni et al., 2002) , METEOR (Banerjee & Lavie, 2005) , ROUGE-L (Lin, 2004) , CIDEr-D (Vedantam et al., 2015) and SPICE (Anderson et al., 2016) . All the metrics are computed by using the codes1 released by COCO Evaluation Server (Chen et al., 2015) .

4.3. COMPARED APPROACHES:

To verify the merit of our LSTM-A models, we compared the following state-of-the-art methods.

(1) NIC & LSTM (Vinyals et al., 2015) : NIC attempts to directly translate from image pixels to natural language with a single deep neural network. The image representations are only injected into LSTM at the initial time step. We directly extract the results reported in (You et al., 2016) and name this run as NIC. Furthermore, for fair comparison, we also include one run LSTM which is our implementation of NIC.

Table 1 : Performance of our proposed models and other state-of-the-art methods on COCO, where B@N , M, R, C and S are short for BLEU@N , METEOR, ROUGE-L, CIDEr-D and SPICE scores. All values are reported as percentage (%).

Model B@1 B@2 B@3 B@4 M R C S NIC (Vinyals et al., 2015) 66 (2) LRCN (Donahue et al., 2015) : LRCN inputs both image representations and previous word into LSTM at each time step for sentence generation.

(3) Hard-Attention (HA) & Soft-Attention (SA) (Xu et al., 2015) : Spatial attention on convolutional features of an image is incorporated into the encoder-decoder framework through two kinds of mechanisms: 1) "hard" stochastic attention mechanism equivalently by reinforce learning and 2) "soft" deterministic attention mechanism with standard back-propagation.

(4) ATT (You et al., 2016) : ATT utilizes attributes as semantic attention to combine image representations and attributes in RNN for image captioning.

(5) Sentence-Condition (SC) (Zhou et al., 2016) (7) CaptionBot (Tran et al., 2016) : CaptionBot is a publicly image captioning system2 which is mainly built on vision models by using Deep residual networks (ResNets) (He et al., 2016) to detect visual concepts, MELM (Fang et al., 2015) language model for sentence generation and DMSM (Fang et al., 2015) for caption ranking. Entity recognition model for celebrities and landmarks is further incorporated to enrich captions and the confidence scoring model is finally utilized to select the output caption.

(8) LSTM-A: LSTM-A 1 , LSTM-A 2 , LSTM-A 3 , LSTM-A 4 , and LSTM-A 5 are five variants derived from our proposed LSTM-A framework. In addition, LSTM-A * is an oracle run that inputs groundtruth attributes in the LSTM-A 3 architecture.

4.4. PERFORMANCE COMPARISON:

Performance on COCO Table 1 shows the performances of different models on COCO image captioning dataset. It is worth noting that the performances of different approaches here are based on different image representations. Specifically, VGG architecture (Simonyan & Zisserman, 2015) is utilized as image feature extractor in the methods of Hard-Attention & Soft-Attention and Sentence-Condition, while GoogleNet (Szegedy et al., 2015) is exploited in NIC, LRCN, ATT, LSTM and our LSTM-A. In view that the GoogleNet and VGG features are comparable, we compare directly with results. Overall, the results across eight evaluation metrics consistently indicate that our proposed LSTM-A exhibits better performance than all the state-of-the-art techniques including non-attention models (NIC, LSTM, LRCN) and attention-based methods (Hard-Attention, Soft-Attention, ATT, Sentence-Condition). In particular, the CIDEr-D can achieve 98.6%, which is to date the high- There is a significant performance gap between ATT and LSTM-A 5 . Though both runs involve the utilization of image representations and attributes, they are fundamentally different in the way that the performance of ATT is as a result of modulating the strength of attention on attributes to the previous words, and LSTM-A 5 is by employing attributes as auxiliary knowledge to complement image representations. This somewhat reveals the weakness of semantic attention model, where the prediction errors will accumulate along the generated sequence.

Compared to LSTM-A 1 , LSTM-A 2 which is augmented by integrating image representations performs better, but the performances are lower than LSTM-A 3 . The results indicate that LSTM-A 3 , in comparison, benefits from the mechanism of first feeding high-level attributes into LSTM instead of starting from inserting image representations in LSTM-A 2 . The chance that a good start point can be attained and lead to performance gain is better. LSTM-A 4 feeding the image representations at each time step yields inferior performances to LSTM-A 3 , which only inputs image representations once. We speculate that this may because the noise in the image can be explicitly accumulated and thus the network overfits more easily. In contrast, the performances of LSTM-A 5 which feeds attributes at each time step show the improvements on LSTM-A 3 . The results demonstrate that the high-level attributes are more accurate and easily translated into human understandable sentence. Among the five proposed LSTM-A architectures, LSTM-A 3 achieves the best performances in terms of BLEU@1 and METEOR, while LSTM-A 5 performs the best in other six evaluation metrics. The performances of the oracle run LSTM-A * could be regarded as the upper bound of employing attributes in our framework and lead to large performance gain against LSTM-A 3 . Such an upper bound enables us to obtain more insights on the factor accounting for the success of the current attribute augmented architecture and also provides guidance to future research in this direction. More specifically, the results, on one hand, indicate the advantage and great potential of leveraging attributes for boosting image captioning, and on the other, suggest that more efforts are further required towards mining and representing attributes more effectively.

Performance on COCO online testing server We also submitted our best run in terms of ME-TEOR, i.e., LSTM-A 3 , to online COCO testing server and evaluated the performance on official testing set. Table 2 shows the performance Leaderboard on official testing image set with 5 reference captions (c5) and 40 reference captions (c40). Please note that here we utilize the outputs of 2,048-way pool5 layer from ResNet-152 as image representations and train the attribute detectors by ResNet-152 in our final submission. Only the latest top-3 performing methods which have been officially published are included in the table. Compared to the top performing methods, our proposed LSTM-A 3 achieves the best performance across all the evaluation metrics on both c5 and c40 testing sets, and ranks the first on the Leaderboard. Ground Truth: a man sitting on a couch holding a wii remote control LSTM-A1: a man sitting on a couch in a living room LSTM-A2: a man sitting on a couch with a laptop LSTM-A3: a man sitting on a couch with a remote LSTM-A4: a woman sitting on a couch with a laptop LSTM-A5: a man sitting on a couch with a remote 

4.5. HUMAN EVALUATION:

To better understand how satisfactory are the sentences generated from different methods, we also conducted a human study to compare our LSTM-A 3 against three approaches, i.e., CaptionBot, L-RCN and LSTM. A total number of 12 evaluators (6 females and 6 males) from different education backgrounds, including computer science (4), business (2), linguistics (2) and engineering (4), are invited and a subset of 1,000 images is randomly selected from testing set for the subjective evaluation. The evaluation process is as follows. All the evaluators are organized into two groups. We show the first group all the four sentences generated by each approach plus the five human-annotated sentences and ask them the question: Do the systems produce captions resembling human-generated sentences? In contrast, we show the second group once only one sentence generated by different approach or human annotation and they are asked: Can you determine whether the given sentence has been generated by a system or by a human being? From evaluators' responses, we calculate two metrics: 1) M1: percentage of captions that are evaluated as better or equal to human caption; 2) M2: percentage of captions that pass the Turing Test. Table 3 lists the result of the user study. Overall, our LSTM-A 3 is clearly the winner for all two criteria. In particular, the percentage achieves 62.8% and 72.2% in terms of M1 and M2, respectively, making the absolute improvement over the best competitor CaptionBot by 4.6% and 5.9%. inputs. Take the first image as an example, the predicted subject is "a cake" in LSTM model. By additionally incorporating the detected attributes, e.g., "candles" and "birthday," the output subject in the sentence by our LSTM-A 3 changes into "a birthday cake with candles," demonstrating the advantage of the auxiliary attribute inputs.

Sentence generation comparison between five LSTM-A architectures. The examples of sentence generated by our five LSTM-A architectures are further illustrated in Figure 3 . In general, the sentences generated by LSTM-A 3 and LSTM-A 5 are very comparable and more accurate than those by LSTM-A 1 , LSTM-A 2 and LSTM-A 4 . For instance, LSTM-A 3 and LSTM-A 5 produce the sentence of "a bunch of stuffed animals hanging from a ceiling," which describes the first image very precisely and finely.

Sentence generation comparison across different approaches. Figure 4 showcases a few sentence examples generated by different methods, the detected high-level attributes, and humanannotated ground truth sentences. From these exemplar results, it is easy to see that all of these automatic methods can generate somewhat relevant sentences, while our proposed LSTM-A 3 can predict more relevant keywords by jointly exploiting high-level attributes and image representations for image captioning. For example, compared to subject term "a group of people" and "a man" in the sentence generated by LSTM and CaptionBot respectively, "a man and a dog" in our LSTM-A 3 is more precise to describe the image content in the first image, since the keyword "dog" is one of the detected attributes and directly injected into LSTM to guide the sentence generation. Similarly, verb term "holding" which is also detected as one high-level attribute presents the fourth image more exactly. Moreover, our LSTM-A 3 can generate more descriptive sentence by enriching the semantics with high-level attributes. For instance, with the detected adjective "red," the generated sentence "a red and white plane flying over a body of water" of the fifth image depicts the image content more comprehensive.

4.7. ANALYSIS OF THE BEAM SIZE k:

In order to analyze the effect of the beam size k in testing stage, we illustrate the performances of our two top performing architectures LSTM-A 3 and LSTM-A 5 with the beam size in the range of {1, 2, 3, 4, 5} in Figure 5 . To make all performances fall into a comparable scale, all scores are normalized by the highest score of each evaluation metric. As shown in Figure 5 , we can see that almost all performances in terms of each evaluation metric are like the "∧" shapes when beam size k varies from 1 to 5. Hence, we set the beam size k as 3 in our experiments, which can achieve the best performance with a relatively small beam size.

:

https://github.com/tylin/coco-caption

https://www.captionbot.ai

Figures and Tables:

Figure 1: Five variants of our LSTM-A framework (better viewed in color).

Figure 2: Visualization of prediction changes w.r.t. the additional attribute inputs. The attributes are predicted by MIL method in (Fang et al., 2015) and the output sentences are generated by LSTM and LSTM-A3.

Figure 3: Sentence examples generated by our five LSTM-A architectures and one ground truth sentence.

LSTM-A3: a person holding a cell phone in their hand Ground Truth: ① a smart phone being held up in front of a lap top ② the person is holding his cell phone while on his laptop ③ someone holding a cell phone in front of a laptop Attributes: computer: 0.764 keyboard: 0.748 screen: 0.627 holding: 0.558 person: 0.515 phone: 0.434 hand: 0.404 remote: 0.395 Generated Sentences: LSTM: a group of people flying kites in the sky CaptionBot: I think it's a plane is flying over the water. LSTM-A3: a red and white plane flying over a body of water Ground Truth: ① a plane with water skies for landing gear coming in for a landing at a lake ② a plane flying through a sky above a lake ③ a red and white plane is flying over some water group of people on a boat in the water CaptionBot: I think it's a man with a small boat in a body of water. LSTM-A3: a man and a dog on a boat in the water Ground Truth: ① an image of a man in a boat with a dog ② a person on a rowboat with a dalmatian dog on the boat ③ old woman rowing a boat with a dog Attributes: snow: 0.996 standing: 0.841 zebra: 0.828 enclosure: 0.629 zoo: 0.492 ground: 0.490 area: 0.417 walking: 0.322 Generated Sentences: LSTM: a zebra is standing in a dirt area CaptionBot: I think it's a zebra is standing in the snow. LSTM-A3: a zebra standing in the snow near a fence Ground Truth: ① one zebra standing in snow near a stone wall ② a zebra is standing in a snowy field ③ a zebra stands in snow in front of a wall Attributes: holding: 0.958 walking: 0.725 people: 0.493 umbrella: 0.548 woman: 0.421 person: 0.361 standing: 0.289 playing: 0.271 Generated Sentences: LSTM: a man walking down a street holding a surfboard CaptionBot: I am not really confident, but I think it's a man walking down a sidewalk holding an umbrella. LSTM-A3: a group of people walking down a street holding umbrellas Ground Truth: ① a couple of kids walking with umbrellas in their hands ② two women walking side by side holding umbrellas ③ an image of two girls walking with umbrellas Attributes: traffic: 0.746 sign: 0.690 street: 0.555 light: 0.446 signs: 0.374 building: 0.344 stop: 0.313 pole: 0.296 Generated Sentences: LSTM: a street sign that is on a pole CaptionBot: I think it's a sign hanging off the side of a building. LSTM-A3: a street sign with a traffic light on it Ground Truth: ① a traffic light in front of some business signs ② a traffic light near a store front ③ a traffic light atop a post in a business district

Figure 4: Attributes and sentences generation results on COCO. The attributes are predicted by MIL method in (Fang et al., 2015) and the output sentences are generated by 1) LSTM, 2) CaptionBot 2 , 3) our LSTM-A3, and 4) Ground Truth: randomly selected three ground truth sentences.

Figure 5: The effect of beam size k on (a) LSTM-A3 and (b) LSTM-A5.

Leaderboard of the published state-of-the-art image captioning models on the online COCO testing server (http://mscoco.org/dataset/#captions-leaderboard), where B@N , M, R, and C are short for BLEU@N , METEOR, ROUGE-L, and CIDEr-D scores. All values are reported as percentage (%).

Table content: <table><tr><td>Model</td><td>c5</td><td>B@1 c40</td><td>c5</td><td>B@2 c40</td><td>c5</td><td>B@3 c40</td><td>c5</td><td>B@4 c40</td><td>c5</td><td>M</td><td>c40</td><td>c5</td><td>R</td><td>c40</td><td>c5</td><td>C</td><td>c40</td></tr><tr><td>MSM@MSRA (LSTM-A 3 )</td><td>75.1</td><td>92.6</td><td>58.8</td><td>85.1</td><td>44.9</td><td>75.1</td><td>34.3</td><td>64.6</td><td>26.6</td><td colspan="2">36.1</td><td>55.2</td><td/><td>70.9</td><td>104.9</td><td colspan="2">105.3</td></tr><tr><td>ATT (You et al., 2016)</td><td>73.1</td><td>90</td><td>56.5</td><td>81.5</td><td>42.4</td><td>70.9</td><td>31.6</td><td>59.9</td><td>25</td><td colspan="2">33.5</td><td>53.5</td><td/><td>68.2</td><td>94.3</td><td/><td>95.8</td></tr><tr><td>Google (Vinyals et al., 2015)</td><td>71.3</td><td>89.5</td><td>54.2</td><td>80.2</td><td>40.7</td><td>69.4</td><td>30.9</td><td>58.7</td><td>25.4</td><td colspan="2">34.6</td><td>53</td><td/><td>68.2</td><td>94.3</td><td/><td>94.6</td></tr><tr><td>MSR Captivator (Devlin et al., 2015)</td><td>71.5</td><td>90.7</td><td>54.3</td><td>81.9</td><td>40.7</td><td>71</td><td>30.8</td><td>60.1</td><td>24.8</td><td colspan="2">33.9</td><td>52.6</td><td/><td>68</td><td>93.1</td><td/><td>93.7</td></tr></table>

The user study using two criteria: M1 -percentage of captions generated by different methods that are evaluated as better or equal to human caption and M2 percentage of captions that pass the Turing Test. METEOR, ROUGR-L, CIDEr-D and SPICE, respectively. The results basically indicate the advantage of exploiting high-level attributes than image representations for image captioning. Furthermore, by additionally incorporating attributes to LSTM model, LSTM-A 2 , LSTM-A 3 and LSTM-A 5 lead to a performance boost, indicating that image representations and attributes are complementary and thus have mutual reinforcement for image captioning. Similar in spirit, LSTM-A 4 improves LRCN by further taking attributes into account.

Table content: <table><tr><td/><td>Human</td><td>LSTM-A3</td><td>CaptionBot</td><td>LSTM</td><td>LRCN</td></tr><tr><td>M1</td><td>-</td><td>62.8</td><td>58.2</td><td>49.2</td><td>43.9</td></tr><tr><td>M2</td><td>90.1</td><td>72.2</td><td>66.3</td><td>57.3</td><td>55.9</td></tr><tr><td colspan="6">est performance reported on COCO dataset when extracting image representations by GoogleNet.</td></tr><tr><td colspan="6">LSTM-A 1 inputting only high-level attributes as representations makes the relative improvemen-</td></tr><tr><td colspan="6">t over LSTM which feeds into image representations instead by 11.6%, 7.8%, 5.1%, 13.9% and</td></tr><tr><td>11.25% in BLEU@4,</td><td/><td/><td/><td/><td/></tr></table>

Visualization of prediction changes w.r.t. the additional attribute inputs. Figure2shows two image examples to illustrate the word prediction changes with respect to the additional attribute

Table content: <table><tr><td/><td>Generated Sentences:</td></tr><tr><td>Attributes:</td><td>Generated Sentences:</td></tr><tr><td>man: 0.669 herd: 0.583 standing: 0.496</td><td/></tr><tr><td>animals: 0.493 walking: 0.471 cows: 0.427</td><td/></tr><tr><td>street: 0.427 road: 0.414</td><td/></tr><tr><td>4.6 QUALITATIVE ANALYSIS</td><td/></tr></table>

Peer review (iclr2017):
CONTRIBUTIONS This paper extends end-to-end CNN+RNN image captioning model with auxiliary attribute predictions. It proposes five variants of network architectures, which take the attribute/image features in alternating orders or at every timestamp. The attributes (i.e., 1,000 most common words on COCO) are obtained by attribute classifiers trained by a multiple instance learning approach. The experiment results indicated that having these attributes as input improves captioning performance on standard metrics, including BLEU, METEOR, ROUGE-L, CIDEr-D. NOVELTY + SIGNIFICANCE All five variants of the network architectures, shown in Fig. 1, have followed a standard seq-to-seq LSTM model. The differences between these variants come from two aspects: 1. the order of image/attribute inputs; 2. whether to input attribute/image features at each time step. No architectural changes have been added to the proposed model over a standard seq-to-seq model. The proposed approach achieves decent performance improvement over previous work, but the technical novelty of this work is significantly limited by existing work that used similar ideas. In particular, Fang et al. 2015 and You et al. 2016 have both used attributes for image captioning. This work has used the same multiple instance learning procedure as Fang et al. 2015 to train visual detectors for common words and used detector outputs as conditional inputs to a language model. In addition, the idea of using image feature as input to every RNN timestamp has been widely explored, for instance, in Donahue et al. 2015. The authors did not offer a clear explanation about the technical contribution of this work over these existing approaches. CLARITY First, it is not clear to me which image features have been used by the baseline methods. As the baselines may rely on different image representations, the experiments would not offer a completely fair comparison. For example, the results of the attention-based models in Table 1 are directly copied from Xu et al., 2015, which were reported with Oxford VGG features, instead of GoogLeNet used by LSTM-A. Even if the baselines do not use the same types of features, it should at least be explicitly mentioned. Besides, the fact that the results of Table 1 and Table 2 are reported with different features (GoogLeNet v.s. ResNet) are not described clearly in the paper. “We select 1,000 most common words on COCO…” How would this approach guarantee that the selected attributes have clear semantic meanings, as many words among the top 1000 would be stop words, abstract nouns, non-visual verbs, etc.? It would be interesting to perform some quantitative analysis to see whether these 1000-dimensional attribute vectors actually carry semantic meanings, or merely capture biases of word distributions from the ground-truth captions. One possible way to justify the importance of semantic attributes is to experiment with ground-truth attributes or attribute classifiers trained on annotated sources such as COCO-Attribute (Patterson et al. 2016) or Visual Genome (Krishna et al. 2016). EXPERIMENTS It would be interesting to analyze how the behavior of the seq-to-seq model changes with respect to the additional attribute inputs. My hypothesis is that the model’s word choices will shift towards words with high scores. This experiments will help us understand how the model can take advantage of the auxiliary attribute inputs. Since the attributes are selected as the most frequent words from COCO, it is likely for the model to overfit the metrics, such as BLEU, that rely on word matching. On the other hand, it has been shown that these automatic caption evaluation metrics do not necessarily correlate with human judgment. Therefore, I think it is necessary to conduct a human study to convince the readers the quality improvement of the proposed model is not caused by overfitting to metrics. SUMMARY This paper demonstrates that image captioning can be improved by having attributes as auxiliary inputs. However, the model has minor novelty given existing work that has explored similar ideas. Besides, more analysis is necessary to demonstrate the semantic meanings of attributes. A human study is recommended to justify the actual performance improvement. Given these points to be improved, I would recommend rejecting this paper in its current form.