Title: Boosting Image Captioning with Attributes
Authors: Ting Yao
Yingwei Pan
Yehao Li
Zhaofan Qiu
Tao Mei
Venue: ICLR 2017
Keywords: Computer vision
Applications

Abstract: Automatically describing an image with a natural language has been an emerging challenge in both fields of computer vision and natural language processing. In this paper, we present Long Short-Term Memory with Attributes (LSTM-A) - a novel architecture that integrates attributes into the successful Convolutional Neural Networks (CNNs) plus Recurrent Neural Networks (RNNs) image captioning framework, by training them in an end-to-end manner. To incorporate attributes, we construct variants of architectures by feeding image representations and attributes into RNNs in different ways to explore the mutual but also fuzzy relationship between them. Extensive experiments are conducted on COCO image captioning dataset and our framework achieves superior results when compared to state-of-the-art deep models. Most remarkably, we obtain METEOR/CIDEr-D of 25.2%/98.6% on testing data of widely used and publicly available splits in \citep{Karpathy:CVPR15} when extracting image representations by GoogleNet and achieve to date top-1 performance on COCO captioning Leaderboard.

TL;DR: Boosting Image Captioning with Attributes

Review discussion:
Can the authors explicitly clarify that no additional supervision was used in the approach? My understanding is that the top words are used as labels and pretrain a set of classifiers, output of which is passed into the LSTM at encoding time.
Also, did the authors filter stop words from the most common words list? I imagine a lot of them are stop words such as "and", "a", and so on. Given that these attributes from Fang et al. are so critical and seem to make so much difference here, it might make sense to copy a short discussion of the approach into this submission. If extra space is needed the related work section is a little too verbose.
Dear Reviewer,
Thanks for your comments.
Yes, no additional supervision is used in our approach. We determine the vocabulary of attributes by using the 1000 most common words in the training captions of COCO dataset and train the attribute detectors by using Multiple Instance Learning (MIL) in [Fang et al., 2015].
No. Following the settings in [Fang et al., 2015], we didn’t filter out any stop words. We will add more details to clarify this in our revision.
Thanks for the suggestions regarding including a short discussion of MIL for attribute learning and shortening the related work section. The suggestions are much appreciated and changes will be made accordingly.
Thanks,
Ting
Hi,
1. I would like to know if you consider doing a human evaluation as the reported metrics are known to not necessarily align with human evolutions?
2. Is it correct that Table 1 does not use ResNet (for your approach), but Table 2 does partially?
3. You write
"In addition, when training the attribute detectors by ResNet-152, our CIDEr-D scores on c5 and c40
testing sets will be further boosted up to 104.9% and 105.3%, respectively."
Why don't you report this in Table 2? What are the other evaluation scores for this variant?
4. Did you consider feeding both, image and attributes at each time step?
Thanks for the clarifications in advance.
Dear Reviewer,
Thanks for your comments.
1. Thanks for suggesting human evaluation. At this time, it is difficult to carry out human evaluation on all the compared approaches, since there is only reported performance but no public model/code available for some of them (e.g., [You et al., 2016]).
Furthermore, manually annotating and comparing the sentences generated by different approaches also requires immense human labor and thus it is hard to be scaled up. Considering this comment, we will conduct additional significance test to verify that our performance boost against different baselines is not by chance. We will include this in our revision.
2. Yes, that is exactly what we did. In Table 1, to ensure the performance of these methods comparable, we learn image representations by GoogleNet.
In contrast, image representations are learnt by ResNet-152 in Table 2. Following the settings in [Fang et al., 2015], attribute representations are learnt by VGG-16 reported in both Table 1&2.
3. Thanks for pointing this out. The performances reported in current Table 2 were officially published in COCO leaderboard when we submitted this paper and our run by training the attribute detectors using ResNet-152 was an internal evaluation on online COCO server at that time. Considering this comment, we will additionally include the performances of this run (c5 [B@1: 75.1%, B@2: 58.8%, B@3: 44.9%, B@4: 34.3%, METEOR: 26.6%, ROUGE-L: 55.2%, CIDEr-D: 104.9%], c40 [B@1: 92.6%, B@2: 85.1%, B@3: 75.1%, B@4: 64.6%, METEOR: 36.1%, ROUGE-L: 70.9%, CIDEr-D: 105.3%]).
4. We indeed experimented with feeding both image and attributes at each time step and that makes a performance drop compared to LSTM-A5. This is similar to the observation that LSTM-A4 yields inferior performances to LSTM-A3. The results basically indicate that the noise in the image can be explicitly accumulated in learning LSTM and leads to overfitting more easily. We will add some discussions to clarify this.
Thanks,
Ting
The paper evaluates different variants to include attributes for caption generation. Attributes are automatically learned from descriptions as in [Fang et al., 2015].
Strength:
1.	The paper discusses and evaluates different variants to incorporate attribute and image representation in a recurrent network.
2.	The paper reports improvements over state-of-the-art on the large-scale MS COCO image caption dataset.
Weaknesses:
1.	The technical novelty of the paper is limited; it is mainly a comparison of different variants to include attributes.
2.
While the exact way how attributes are used is different to prior work, the presented variants are not especially exiting.
3.	The reported metrics are known to not always correlate very well with human judgments.
3.1.	It would be good to additionally also report the SPICE (Anderson et al ECCV 2016) metric, which has shown good correlation w.r.t. human judgments.
3.2.
In contrast to the arguments of the authors during the discussion period, I believe a human evaluation is feasible at least on a subset of the test set. In my experience, most authors will share their generated sentences if they did not publish them.
Also, a comparison between the different variants should be possible.
4.	Qualitative results:
4.1.	How do the different variants A1 to A5 compare? Is there a difference in the sentence between the different approaches?
Other (minor/discussion points)
-	Page 8: “is benefited” -> benefits
Summary:
While the paper presents a valuable experimental evaluation with convincing results, the contribution w.r.t. to novelty and approach is limited.
We would like to thank all the reviewers for your great efforts on our submission. We have tried our best to comply with your valuable recommendations and have made corresponding revisions in the revised manuscript. We feel that the revisions have enhanced the readability of the paper.
R3.1 Significance and Novelty
The main idea of this paper is to explore how to incorporate high-level image attributes for boosting image captioning. We do believe that this is the right way to leverage more knowledge for building richer representations and description models.
In particular, the utilization of attributes has a great potential to be an elegant solution of generating open-vocabulary sentences, making image captioning system really practical and helpful for robotic vision or visually impaired people.
Though the techniques of attribute detector training by multiple instance learning (Fang et al., 2015) or CNN plus RNN captioning models (e.g., Donahue et al., 2015) are studied before, how to integrate the attributes into CNN plus RNN captioning framework is still a problem not yet fully understood in the literature.
Therefore, the main contribution of our work is the proposal of a general image captioning framework with attributes, which is later regarded as an extended direction to the basic captioning model in (Liu et al., 2016)*. More importantly, we devise five variants of attribute augmented architectures, empirically verify the merit of each and provide a comprehensive comparison in between. We do think our work takes a further step forward to enhance image captioning and could have a direct impact of indicating a new future direction of vision and language research.
We have added more explanations about the significance and novelty of our work throughout the paper in our revision.
*: Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama and Kevin Murphy. Optimization of image description metrics using policy gradient methods. arXiv preprint arXiv: 1612.00370v2, 2016.
R3.2 While the exact way how attributes are used is different to prior work, the presented variants are not especially exiting
Thanks. In order to examine the effectiveness of our architectures irrespective of the learnt attributes influence, we conducted an additional experiment by inputting ground-truth attributes in our architectures. In this case, the performances of LSTM-A3 achieve B@1: 95.7%, B@2: 82.5%, B@3: 68.5%, B@4: 55.9%, METEOR: 34.1%, ROUGE-L: 67.3% and CIDEr-D: 150.5%, which could be regarded as the upper bound of employing attributes in our framework and lead to large performance gain against LSTM-A3. The results, on one hand, indicate the advantage and great potential of our proposed architecture for boosting image captioning with attributes, and on the other, suggest that more efforts are further required towards mining and representing attributes more effectively. We have updated Table 1 and added these explanations in our revision.
R3.3 SPICE
Thanks. We have added SPICE as another evaluation metric and reported the SPICE performances in Table 1. The performance trends of SPICE are similar with that of METEOR.
R3.4 Human judgment
Thanks for your valuable suggestions. We conducted an additional human study to compare our LSTM-A3 against three approaches, i.e., CaptionBot, LRCN and LSTM. A total number of 12 evaluators (6 females and 6 males) from different education backgrounds, including computer science (4), business (2), linguistics (2) and engineering (4), are invited and a subset of 1,000 images is randomly selected from testing set for the subjective evaluation. The evaluation process is as follows. All the evaluators are organized into two groups. We show the first group all the four sentences generated by each approach plus the five human-annotated sentences and ask them the question: Do the systems produce captions resembling human-generated sentences?
In contrast, we show the second group once only one sentence generated by different approach or human annotation and they are asked: Can you determine whether the given sentence has been generated by a system or by a human being? From evaluators' responses, we calculate two metrics: 1) M1: percentage of captions that are evaluated as better or equal to human caption; 2) M2: percentage of captions that pass the Turing Test. Table 3, in our revision, lists the result of the user study.
Overall, our LSTM-A3 is clearly the winner for all two criteria.
In particular, the percentage achieves 62.8% and 72.2% in terms of M1 and M2, respectively, making the absolute improvement over the best competitor CaptionBot by 4.6% and 5.9%. We have added the human judgment in our revision.
R3.5 Qualitative analysis between LSTM-A1 to A5
Thanks for this point. We have added one figure (Figure 3 in our revision) to showcase the sentence generated by our proposed five variants. In general, the sentences generated by LSTM-A3 and LSTM-A5 are very comparable and more accurate than those by LSTM-A1, LSTM-A2 and LSTM-A4.
For instance, LSTM-A3 and LSTM-A5 produce the sentence of "a bunch of stuffed animals hanging from a ceiling," which describes the first image very precisely and finely.
The additional results and experiments strengthen the paper as an experimental evaluation paper.
I appreciate the human evaluation and the additional qualitative results.
However, I remain with my assessment and in line with the other reviewers that the technical novelty is not especially big.
My final score is thus somewhere between 6 and 7.
The authors take a standard image captioning system and, in addition to the image, also condition the caption on a fixed vector of attributes. These attributes are the top 1,000 most common words trained with MIL as seen in Fang et al. 2015. The authors investigate 5 ways of plugging in the attributes vector for each image. More or less all of them turn out to work comparably (24 - 25.2 METEOR), but also a good amount better than a standard image captioning model (25.2 > 23.1 METEOR). At the time this approach was state of the art on the MS COCO leaderboard.
I am conflicted judging these kinds of application-heavy papers. It is clear that the technical execution is done relatively well, but there is little to take away or learn. I find it interesting and slightly surprising that you can boost image captioning results by appending these automatically-extracted attributes, but on the topic of how many people would find this relevant or how much additional work it can inspire I am not so sure.
We would like to thank all the reviewers for your great efforts on our submission. We have tried our best to comply with your valuable recommendations and have made corresponding revisions in the revised manuscript. We feel that the revisions have enhanced the readability of the paper.
R2.1 Significance and Novelty
The main idea of this paper is to explore how to incorporate high-level image attributes for boosting image captioning. We do believe that this is the right way to leverage more knowledge for building richer representations and description models.
In particular, the utilization of attributes has a great potential to be an elegant solution of generating open-vocabulary sentences, making image captioning system really practical and helpful for robotic vision or visually impaired people.
Though the techniques of attribute detector training by multiple instance learning (Fang et al., 2015) or CNN plus RNN captioning models (e.g., Donahue et al., 2015) are studied before, how to integrate the attributes into CNN plus RNN captioning framework is still a problem not yet fully understood in the literature.
Therefore, the main contribution of our work is the proposal of a general image captioning framework with attributes, which is later regarded as an extended direction to the basic captioning model in (Liu et al., 2016)*. More importantly, we devise five variants of attribute augmented architectures, empirically verify the merit of each and provide a comprehensive comparison in between. We do think our work takes a further step forward to enhance image captioning and could have a direct impact of indicating a new future direction of vision and language research.
We have added more explanations about the significance and novelty of our work throughout the paper in our revision.
*: Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama and Kevin Murphy. Optimization of image description metrics using policy gradient methods. arXiv preprint arXiv: 1612.00370v2, 2016.
CONTRIBUTIONS
This paper extends end-to-end CNN+RNN image captioning model with auxiliary attribute predictions. It proposes five variants of network architectures, which take the attribute/image features in alternating orders or at every timestamp. The attributes (i.e., 1,000 most common words on COCO) are obtained by attribute classifiers trained by a multiple instance learning approach. The experiment results indicated that having these attributes as input improves captioning performance on standard metrics, including BLEU, METEOR, ROUGE-L, CIDEr-D.
NOVELTY + SIGNIFICANCE
All five variants of the network architectures, shown in Fig. 1, have followed a standard seq-to-seq LSTM model. The differences between these variants come from two aspects: 1. the order of image/attribute inputs; 2. whether to input attribute/image features at each time step. No architectural changes have been added to the proposed model over a standard seq-to-seq model. The proposed approach achieves decent performance improvement over previous work, but the technical novelty of this work is significantly limited by existing work that used similar ideas.
In particular, Fang et al. 2015 and You et al. 2016 have both used attributes for image captioning. This work has used the same multiple instance learning procedure as Fang et al. 2015 to train visual detectors for common words and used detector outputs as conditional inputs to a language model.
In addition, the idea of using image feature as input to every RNN timestamp has been widely explored, for instance, in Donahue et al. 2015. The authors did not offer a clear explanation about the technical contribution of this work over these existing approaches.
CLARITY
First, it is not clear to me which image features have been used by the baseline methods. As the baselines may rely on different image representations, the experiments would not offer a completely fair comparison.
For example, the results of the attention-based models in Table 1 are directly copied from Xu et al., 2015, which were reported with Oxford VGG features, instead of GoogLeNet used by LSTM-A.
Even if the baselines do not use the same types of features, it should at least be explicitly mentioned.
Besides, the fact that the results of Table 1 and Table 2 are reported with different features (GoogLeNet v.s. ResNet) are not described clearly in the paper.
“We select 1,000 most common words on COCO…” How would this approach guarantee that the selected attributes have clear semantic meanings, as many words among the top 1000 would be stop words, abstract nouns, non-visual verbs, etc.? It would be interesting to perform some quantitative analysis to see whether these 1000-dimensional attribute vectors actually carry semantic meanings, or merely capture biases of word distributions from the ground-truth captions. One possible way to justify the importance of semantic attributes is to experiment with ground-truth attributes or attribute classifiers trained on annotated sources such as COCO-Attribute (Patterson et al. 2016) or Visual Genome (Krishna et al. 2016).
EXPERIMENTS
It would be interesting to analyze how the behavior of the seq-to-seq model changes with respect to the additional attribute inputs. My hypothesis is that the model’s word choices will shift towards words with high scores. This experiments will help us understand how the model can take advantage of the auxiliary attribute inputs.
Since the attributes are selected as the most frequent words from COCO, it is likely for the model to overfit the metrics, such as BLEU, that rely on word matching.
On the other hand, it has been shown that these automatic caption evaluation metrics do not necessarily correlate with human judgment.
Therefore, I think it is necessary to conduct a human study to convince the readers the quality improvement of the proposed model is not caused by overfitting to metrics.
SUMMARY
This paper demonstrates that image captioning can be improved by having attributes as auxiliary inputs.
However, the model has minor novelty given existing work that has explored similar ideas.
Besides, more analysis is necessary to demonstrate the semantic meanings of attributes. A human study is recommended to justify the actual performance improvement. Given these points to be improved, I would recommend rejecting this paper in its current form.
We would like to thank all the reviewers for your great efforts on our submission. We have tried our best to comply with your valuable recommendations and have made corresponding revisions in the revised manuscript. We feel that the revisions have enhanced the readability of the paper.
R1.1 Significance and Novelty
The main idea of this paper is to explore how to incorporate high-level image attributes for boosting image captioning. We do believe that this is the right way to leverage more knowledge for building richer representations and description models.
In particular, the utilization of attributes has a great potential to be an elegant solution of generating open-vocabulary sentences, making image captioning system really practical and helpful for robotic vision or visually impaired people.
Though the techniques of attribute detector training by multiple instance learning (Fang et al., 2015) or CNN plus RNN captioning models (e.g., Donahue et al., 2015) are studied before, how to integrate the attributes into CNN plus RNN captioning framework is still a problem not yet fully understood in the literature.
Therefore, the main contribution of our work is the proposal of a general image captioning framework with attributes, which is later regarded as an extended direction to the basic captioning model in (Liu et al., 2016)*. More importantly, we devise five variants of attribute augmented architectures, empirically verify the merit of each and provide a comprehensive comparison in between. We do think our work takes a further step forward to enhance image captioning and could have a direct impact of indicating a new future direction of vision and language research.
We have added more explanations about the significance and novelty of our work throughout the paper in our revision.
*: Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama and Kevin Murphy. Optimization of image description metrics using policy gradient methods. arXiv preprint arXiv: 1612.00370v2, 2016.
R1.2 Image features
Thanks for pointing this out. Yes, the performances of different approaches reported in Table 1 are based on different image representations.
Specifically, Oxford VGG architecture is utilized as image feature extractor in the methods of Hard-Attention & Soft-Attention (Xu et al., 2015) and Sentence-Condition (Zhou et al., 2016), while GoogleNet is exploited in NIC & LSTM (Vinyals et al., 2015), LRCN (Donahue et al., 2015), ATT (You et al., 2016) and our LSTM-A. In view that the GoogleNet and Oxford VGG features are comparable, we compare directly with results in Table 1. Considering the reviewer’s comments, we will explicitly indicate the image features exploited in different approaches.
In addition, we utilize ResNet-152 as image feature extractor in our submission to the online testing server and report the results in Table 2. We have rephrased the statement to better describe the image features in our revision.
R1.3 Experiments with ground-truth attributes
Thanks for your valuable suggestions. We conducted an additional experiment by inputting ground-truth attributes in our LSTM-A3 architecture. In this case, the performances achieve B@1: 95.7%, B@2: 82.5%, B@3: 68.5%, B@4: 55.9%, METEOR: 34.1%, ROUGE-L: 67.3% and CIDEr-D: 150.5%, which could be regarded as the upper bound of employing attributes in our framework and lead to large performance gain against LSTM-A3. Such an upper bound enables us to obtain more insights on the factor accounting for the success of the current attribute augmented architecture and also provides guidance to future research in this direction. More specifically, the results, on one hand, indicate the advantage and great potential of leveraging attributes for boosting image captioning, and on the other, suggest that more efforts are further required towards mining and representing attributes more effectively. We have updated Table 1 and added these explanations in our revision.
R1.4 Visualization of prediction changes with respect to the additional attribute inputs
Thanks. We have added two image examples to clarify the prediction changes with respect to the additional attribute inputs in Figure 2. Take the first image as an example, the predicted subject is "a cake" in LSTM model. By additionally incorporating the detected attributes, e.g., "candles" and "birthday," the output subject in the sentence by our LSTM-A changes into "a birthday cake with candles," demonstrating the advantage of the auxiliary attribute inputs.
R1.5 Human judgment
Thanks for your valuable suggestions. We conducted an additional human study to compare our LSTM-A3 against three approaches, i.e., CaptionBot, LRCN and LSTM. A total number of 12 evaluators (6 females and 6 males) from different education backgrounds, including computer science (4), business (2), linguistics (2) and engineering (4), are invited and a subset of 1,000 images is randomly selected from testing set for the subjective evaluation. The evaluation process is as follows. All the evaluators are organized into two groups. We show the first group all the four sentences generated by each approach plus the five human-annotated sentences and ask them the question: Do the systems produce captions resembling human-generated sentences?
In contrast, we show the second group once only one sentence generated by different approach or human annotation and they are asked: Can you determine whether the given sentence has been generated by a system or by a human being? From evaluators’ responses, we calculate two metrics: 1) M1: percentage of captions that are evaluated as better or equal to human caption; 2) M2: percentage of captions that pass the Turing Test. Table 3, in our revision, lists the result of the user study.
Overall, our LSTM-A3 is clearly the winner for all two criteria.
In particular, the percentage achieves 62.8% and 72.2% in terms of M1 and M2, respectively, making the absolute improvement over the best competitor CaptionBot by 4.6% and 5.9%. We have added the human judgment in our revision.

Meta-review:
role: PC
data: comment: The paper investigates several ways of conditioning an image captioning model on image attributes. While the results are good, the novelty is too limited for the paper to be accepted.

Decision: Reject