Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper takes the MSCOCO captioning dataset, heuristically parses the sentences into Subject Relation Object (SRO) triplets similar to Farhadi et al 2010, and then evaluates the Show Attend and Tell and some baselines on two splits of the data: 1) a split where the Test set has the same SRO distribution but new images (standard setting) and 2) a split where the Test set contains SRO triplets where the individual pieces were seen in the training data, but the particular composition wasn’t. The paper shows that the generalization performance is very poor on (2), almost as bad as a relatively weak SSVM baseline.

The paper is fairly well written, modulo some odd quirks like undefined terms, though I have some larger recommendations. First, this is not a model paper, it’s more of a dataset/evaluation paper. Therefore, the related work should focus specifically on this aspect instead of broadly discussing the modeling approaches used in image captioning. Second, the description of the main SA&T model is slightly too short and it’s not clear how the SRO triplets are predicted. Is the LSTM emitting S,R,O in that order, pretending these are sentences of 3 words? The decoders are also using soft attention. I think I know what that means, but this could be clarified.

In terms of the results, at some level I’m not too surprised about the results because the LSTM is trained to model the joint distribution over SRO triplets, so it might be reluctant to predict e.g. an O given S,R that never occurred in the training data. Of course, one would like the model to learn the true underlying function from images to labels without relying on the more shallow dependencies of the labels, and it appears that the model struggles here, for this many datapoints. This has been qualitatively noted in the literature before, but I’m not aware of anyone who studied this more quantitatively and the related work doesn’t cover this well. For example, VQA also struggles with this problem, which also partly motivated VQA 2.0. I’m also aware of CLEVR, which proposed to study these problems in a synthetic setting, etc. Another interesting baseline here might be the same SA&T model, but feeding in all zero images, which would be a more controlled comparison, to see just how much the images are used, and to what extent it’s just the joint distribution modeling.

In conclusions, this is one approach to measuring a problem that has been noted by the community. The SA&T model and a few very simple baselines are evaluated, but the conclusion isn’t particularly striking and the analysis isn’t particularly deep and therefore there is not too much to take away.
review_title: A good target for further study but limited execution

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies how much compositionality approaches for image captioning have. The experiments compare two scenarios: on the standard COCO test split (which has similar compositions, but novel images), and on a new split which does not contain known components in a novel composition. Specifically, the paper parses coco captions into triplets of subject – relation – object and evaluates these rather than actual captions.

Strength: 
-	The paper has a clear exposition and motivation.
-	The paper studies an interesting scenario of how compositional approaches are for image captioning.

Main Weaknesses:
1.	The result suggest that the studied approaches/baselines cannot handle the scenario of novel triplets, i.e. have low compositionality. According to the authors expositions this was already known beforehand.
2.	The specific experimental setup does not allow to study how this affects the actual generation of sentence, e.g. there might be some approaches which are very good at novel triplet prediction, but do not solve the problem of sentence generation.
3.	The paper does not propose any novel approach and only studies *one* captioning approach (Show attend and tell), although the text sometimes suggests that there are multiple.
4.	From a learning perspective the studied model, show-attend-and-tell, does exactly what it is trained for: It predicts very low probability for tuples which have very low joint probability according to the training data. The LSTM learns exactly this aspect [similar the pair-wise terms in the SSVM model]. If one does not want to exploit the joint probabilities, one should just look at the unaries.
5.	The paper misses to provide any insight how this know problem could be approached.
6.	Please also provide precision@k for each of the components of, i.e. separately for S, R, and O to understand better where the problem of joint task originates from (in both settings: coco split and compositional split).



Further Weaknesses
7.	It would be interesting and important to know, how well a model does which does not have pair-wise probabilities, e.g. only f_S, f_O, and f_R, but not f_SR and f_RO.
8.	Please provide a better definition how precision@k is computed: does it mean if for @1 that it is on one if the highest ranked triplet matches *any* of the ground truth triplets?
9.	Space can probably be saved by removing Figure 2, and only reporting @1, @5, @10 in a table. 
10.	Would the precision@k evaluation not be better to be cumulative?
11.	What does the “Conv” stand for in SSVM/Conv?
12.	Last sentence Sec. 3.3: Does this really apply only to SSVM or to all models?
13.	Why has the MF 0% on the coco split? What percentage does it have on the training coco split? If it really is always very low, maybe this can be removed to save space, and only mention this at one point in the text.
14.	Section 1: what is “open-IE”?
15. Please cite the actual publications not the arXives, whenever available.



Conclusion:
While the approached problem is interesting and relevant, the paper does not propose any novel approach, but rather examines only a single captioning approach with a negative result and no conclusion where to go from here.
Combined with many unclarities mentioned above, I lean towards rejecting this paper.

review_title: Interesting direction, but limited novelty and conclusion

Decision: Reject