Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: 1) One of the nice advantages of models with attention is added interpretability. Do you have any interesting examples of failure in your model?
2) Although your visualization method is pretty consistent with work on this type of attention model, do you have comments on how valid your visualization technique of rescaling is for your particular model? 


review_title: questions

final_score: rating: 7: Good paper, accept
review_content: The paper presents an architecture to incrementally attend to image regions - at multiple layers of a deep CNN. In contrast to most other models, the model does not apply a weighted average pooling in the earlier layers of the network but only in the last layer. Instead, the features are reweighted in each layer with the predicted attention.

1.	Contribution of approach: The approach to use attention in this way is to my knowledge novel and interesting.
2.	Qualitative results: 
2.1.	I like the large number of qualitative results; however, I would have wished the focus would have been less on the “number” dataset and more on the Visual Genome dataset.
2.2.	The qualitative results for the Genome dataset unfortunately does not provide the predicted attributes. It would be interesting to see e.g. the highest predicted attributes for a given query. So far the results only show the intermediate results.
3.	Qualitative results:
3.1.	The paper presents results on two datasets, one simulated dataset as well as Visual Genome. On both it shows moderate but significant improvements over related approaches.
3.2.	For the visual genome dataset, it would be interesting to include a quantitative evaluation how good the localization performance is of the attention approach.
3.3.	It would be interesting to get a more detailed understanding of the model by providing results for different CNN layers where the attention is applied.
4.	It would be interesting to see results on more established tasks, e.g. VQA, where the model should similarly apply. In fact, the task on the numbers seems to be identical to the VQA task (input/output), so most/all state-of-the-art VQA approaches should be applicable.


Other (minor/discussion points)
-	Something seems wrong in the last two columns in Figure 11: the query “7” is blue not green. Either the query or the answer seem wrong.
-	Section 3: “In each layer, the each attended feature map” -> “In each layer, each attended feature map”
-	I think Appendix A would be clearer if it would be stated that is the attention mechanism used in SAN and which work it is based on.


Summary:
While the experimental evaluation could be improved with more detailed evaluation, comparisons, and qualitative results, the presented evaluation is sufficient to validate the approach. The approach itself is novel and interesting to my knowledge and speaks for acceptance.

review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes an attention mechanism which is essentially a gating on every spatial feature. Though they claim novelty through the attention being progressive, progressive attention has been done before [Spatial Transformer Networks, Deep Networks with Internal Selective Attention through Feedback Connections], and the element-wise multiplicative gates are very similar to convolutional LSTMs and Highway Nets. There is a lack of novelty and no significant results.

Pros:
- The idea of progressive attention on features is good, but has been done in [Spatial Transformer Networks, Deep Networks with Internal Selective Attention through Feedback Connections]
- Good visualisations.

Cons:
- No progressive baselines were evaluated, e.g. STN and HAN at every layer acting on featuremaps.
- Not clear how the query is fed into the localisation networks of baselines.
- The difference in performance between author-made synthetic data and the Visual Genome datasets between baselines and PAN is very different. Why is this? There is no significant performance gain on any standard datasets.
- No real novelty.
review_title: null

final_score: rating: null
review_content: Hi,

while I don't believe this will work better: 
Did you try to keep the intermediate layers fixed and only trained the attention?

Thanks.
review_title: Clarification

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a hierarchical attention model that uses multiple stacked layers of soft attention in a convnet. The authors provide results on a synthetic dataset in addition to doing attribute prediction on the Visual Genome dataset.

Overall I think this is a well executed paper, with good experimental results and nice qualitative visualizations. The main thing I believe it is missing would be experiments on a dataset like VQA which would help better place the significance of this work in context of other approaches.  

An important missing citation is Graves 2013 which had an early version of the attention model. 

Minor typo:
"It confins possible attributes.." -> It confines..
"ImageNet (Deng et al., 2009), is used, and three additional" -> ".., are used,"
review_title: Good paper, but would help to have experiments on a more benchmarked dataset

Meta-review: The program committee appreciates the authors' response to concerns raised in the reviews. Authors have conducted additional experiments and provided comparisons to other existing models. However, reviewer scores are not leaning sufficiently towards acceptance.
 
 The effectiveness of this approach on realistic data still remains unclear in the context of existing approaches. I agree that the reported improvement on Visual Genome over the baseline is non-trivial. But evaluating an existing state-of-the-art VQA approach (for instance) would help better place the performance of this approach in perspective relative to state-of-the-art. 
 
 Incorporating reviewer comments, and more convincing demonstration of the model's capabilities on realistic data will help make the paper stronger.
Decision: Reject