Venue: ICLR 2017 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
The authors take a standard image captioning system and, in addition to the image, also condition the caption on a fixed vector of attributes. These attributes are the top 1,000 most common words trained with MIL as seen in Fang et al. 2015. The authors investigate 5 ways of plugging in the attributes vector for each image. More or less all of them turn out to work comparably (24 - 25.2 METEOR), but also a good amount better than a standard image captioning model (25.2 > 23.1 METEOR). At the time this approach was state of the art on the MS COCO leaderboard.

I am conflicted judging these kinds of application-heavy papers. It is clear that the technical execution is done relatively well, but there is little to take away or learn. I find it interesting and slightly surprising that you can boost image captioning results by appending these automatically-extracted attributes, but on the topic of how many people would find this relevant or how much additional work it can inspire I am not so sure.

review_title: good results, very limited novelty

final_score: rating: null
review_content: Hi,

1. I would like to know if you consider doing a human evaluation as the reported metrics are known to not necessarily align with human evolutions?

2. Is it correct that Table 1 does not use ResNet (for your approach), but Table 2 does partially?

3. You write
"In addition, when training the attribute detectors by ResNet-152, our CIDEr-D scores on c5 and c40
testing sets will be further boosted up to 104.9% and 105.3%, respectively." 
Why don't you report this in Table 2? What are the other evaluation scores for this variant?

4. Did you consider feeding both, image and attributes at each time step?


Thanks for the clarifications in advance.
review_title: Clarifications

final_score: rating: 4: Ok but not good enough - rejection
review_content: CONTRIBUTIONS
This paper extends end-to-end CNN+RNN image captioning model with auxiliary attribute predictions. It proposes five variants of network architectures, which take the attribute/image features in alternating orders or at every timestamp. The attributes (i.e., 1,000 most common words on COCO) are obtained by attribute classifiers trained by a multiple instance learning approach. The experiment results indicated that having these attributes as input improves captioning performance on standard metrics, including BLEU, METEOR, ROUGE-L, CIDEr-D. 

NOVELTY + SIGNIFICANCE
All five variants of the network architectures, shown in Fig. 1, have followed a standard seq-to-seq LSTM model. The differences between these variants come from two aspects: 1. the order of image/attribute inputs; 2. whether to input attribute/image features at each time step. No architectural changes have been added to the proposed model over a standard seq-to-seq model. The proposed approach achieves decent performance improvement over previous work, but the technical novelty of this work is significantly limited by existing work that used similar ideas. In particular, Fang et al. 2015 and You et al. 2016 have both used attributes for image captioning. This work has used the same multiple instance learning procedure as Fang et al. 2015 to train visual detectors for common words and used detector outputs as conditional inputs to a language model. In addition, the idea of using image feature as input to every RNN timestamp has been widely explored, for instance, in Donahue et al. 2015. The authors did not offer a clear explanation about the technical contribution of this work over these existing approaches.

CLARITY
First, it is not clear to me which image features have been used by the baseline methods. As the baselines may rely on different image representations, the experiments would not offer a completely fair comparison. For example, the results of the attention-based models in Table 1 are directly copied from Xu et al., 2015, which were reported with Oxford VGG features, instead of GoogLeNet used by LSTM-A. Even if the baselines do not use the same types of features, it should at least be explicitly mentioned. Besides, the fact that the results of Table 1 and Table 2 are reported with different features (GoogLeNet v.s. ResNet) are not described clearly in the paper.

“We select 1,000 most common words on COCO…” How would this approach guarantee that the selected attributes have clear semantic meanings, as many words among the top 1000 would be stop words, abstract nouns, non-visual verbs, etc.? It would be interesting to perform some quantitative analysis to see whether these 1000-dimensional attribute vectors actually carry semantic meanings, or merely capture biases of word distributions from the ground-truth captions. One possible way to justify the importance of semantic attributes is to experiment with ground-truth attributes or attribute classifiers trained on annotated sources such as COCO-Attribute (Patterson et al. 2016) or Visual Genome (Krishna et al. 2016).

EXPERIMENTS
It would be interesting to analyze how the behavior of the seq-to-seq model changes with respect to the additional attribute inputs. My hypothesis is that the model’s word choices will shift towards words with high scores. This experiments will help us understand how the model can take advantage of the auxiliary attribute inputs.

Since the attributes are selected as the most frequent words from COCO, it is likely for the model to overfit the metrics, such as BLEU, that rely on word matching. On the other hand, it has been shown that these automatic caption evaluation metrics do not necessarily correlate with human judgment. Therefore, I think it is necessary to conduct a human study to convince the readers the quality improvement of the proposed model is not caused by overfitting to metrics.

SUMMARY
This paper demonstrates that image captioning can be improved by having attributes as auxiliary inputs. However, the model has minor novelty given existing work that has explored similar ideas. Besides, more analysis is necessary to demonstrate the semantic meanings of attributes. A human study is recommended to justify the actual performance improvement. Given these points to be improved, I would recommend rejecting this paper in its current form.
review_title: Review: Boosting Image Captioning with Attributes

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper evaluates different variants to include attributes for caption generation. Attributes are automatically learned from descriptions as in [Fang et al., 2015].  


Strength:
1.	The paper discusses and evaluates different variants to incorporate attribute and image representation in a recurrent network.
2.	The paper reports improvements over state-of-the-art on the large-scale MS COCO image caption dataset.

Weaknesses:
1.	The technical novelty of the paper is limited; it is mainly a comparison of different variants to include attributes.
2.	While the exact way how attributes are used is different to prior work, the presented variants are not especially exiting.
3.	The reported metrics are known to not always correlate very well with human judgments.
3.1.	It would be good to additionally also report the SPICE (Anderson et al ECCV 2016) metric, which has shown good correlation w.r.t. human judgments.
3.2.	In contrast to the arguments of the authors during the discussion period, I believe a human evaluation is feasible at least on a subset of the test set. In my experience, most authors will share their generated sentences if they did not publish them. Also, a comparison between the different variants should be possible.
4.	Qualitative results:
4.1.	How do the different variants A1 to A5 compare? Is there a difference in the sentence between the different approaches?


Other (minor/discussion points)
-	Page 8: “is benefited” -> benefits


Summary:
While the paper presents a valuable experimental evaluation with convincing results, the contribution w.r.t. to novelty and approach is limited.

review_title: Review

final_score: rating: null
review_content: Can the authors explicitly clarify that no additional supervision was used in the approach? My understanding is that the top words are used as labels and pretrain a set of classifiers, output of which is passed into the LSTM at encoding time. Also, did the authors filter stop words from the most common words list? I imagine a lot of them are stop words such as "and", "a", and so on. Given that these attributes from Fang et al. are so critical and seem to make so much difference here, it might make sense to copy a short discussion of the approach into this submission. If extra space is needed the related work section is a little too verbose.


review_title: questions

Meta-review: The paper investigates several ways of conditioning an image captioning model on image attributes. While the results are good, the novelty is too limited for the paper to be accepted.
Decision: Reject