Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposed an end-to-end trainable hierarchical attention mechanism for CNN. The proposed method computes 2d spatial attention map at multiple layers in CNN, where each attention map is obtained by computing compatibility scores between the intermediate features and the global feature. The proposed method demonstrated noticeable performance improvement on various discriminative tasks over existing approaches. 

Overall, the idea presented in the paper is simple yet solid, and showed good empirical performance. The followings are several concerns and suggestions.  

1. The authors claimed that this is the first end-to-end trainable hierarchical attention model, but there is a previous work that also addressed the similar task:
Seo et al, Progressive Attention Networks for Visual Attribute Prediction, in Arxiv preprint:1606.02393, 2016 

2. The proposed attention mechanism seems to be fairly domain (or task ) specific, and may not be beneficial for strong generalization (generalization over unseen category). Since this could be a potential disadvantage, some discussions or empirical study on cross-category generalization seems to be interesting.

3. The proposed attention mechanism is mainly demonstrated for single-class classification task, but it would be interesting to see if it can also help the multi-class classification (e.g. image classification on MS-COCO or PASCAL VOC datasets)

4. The localization performance of the proposed attention mechanism is evaluated by weakly-supervised semantic segmentation tasks. In that perspective, it would be interesting to see the comparisons against other attention mechanisms (e.g. Zhou et al 2016) in terms of localization performance.

review_title: No title

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes an end-to-end trainable attention module, which takes as input the 2D feature vector map and outputs a 2D matrix of scores for each map. The goal is to make the learned attention maps highlight the regions of interest while suppressing background clutter. Experiments conducted on image classification and weakly supervised segmentation show the effectiveness of the proposed method.

Strength of this paper:
1) Most previous work are all implemented as post-hoc additions to fully trained networks while this work is end-to-end trainable. Not only the newly added weights for attention will be learned, so are the original weights in the network.
2) The generalization ability shown in Table 3 is very good, outperforming other existing network by a large margin.
3) Visualizations shown in the paper are convincing. 

Some weakness:
1) Some of the notations are unclear in this paper, vector should be bold, hard to differentiate vector and scalar.
2) In equation (2), l_i and g should have different dimensionality, how does addition work? Same as equation (3)
3) The choice of layers to add attention modules is unclear to me. The authors just pick three layers from VGG to add attention, why picking those 3 layers? Is it better to add attention to lower layers or higher layers? Why is it the case that having more layers with attention achieves worse performance?

review_title: Results are good, some unclear explanation 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a network with the standard soft-attention mechanism for classification tasks, where the global feature is used to attend on multiple feature maps of local features at different intermediate layers of CNN. The attended features at different feature maps are then used to predict the final classes by either concatenating features or ensembling results from individual attended features. The paper shows that the proposed model outperforms the baseline models in classification and weakly supervised segmentation.

Strength:
- It is interesting idea to use the global feature as a query in the attention mechanism while classification tasks do not naturally involve a query unlike other tasks such as visual question answering and image captioning.

- The proposed model shows superior performances over GAP in multiple tasks.

Weakness:
- There are a lot of missing references. There have been a bunch of works using the soft-attention mechanism in many different applications including visual question answering [A-C], attribute prediction [D], image captioning [E,F] and image segmentation [G]. Only two previous works using the soft-attention (Bahdanau et al., 2014; Xu et al., 2015) are mentioned in Introduction but they are not discussed while other types of attention models (Mnih et al., 2014; Jaderberg et al., 2015) are discussed more.

- Section 2 lacks discussions about related work but is more dedicated to emphasizing the contribution of the paper.

- The global feature is used as the query vector for the attention calculation. Thus, if the global feature contains information for a wrong class, the attention quality should be poor too. Justification on this issue can improve the paper.

- [H] reports the performance on the fine-grained bird classification using different type of attention mechanism. Comparison and justification with this method can improve the paper. The performance in [H] is almost 10 % point higher accuracy than the proposed model.

- In the segmentation experiments, the models are trained on extremely small images, which is unnatural in segmentation scenarios. Experiments on realistic settings should be included. Moreover, [G] introduces a method of using an attention model for segmentation, while the paper does not contain any discussion about it.


Overall, I am concerned that the proposed model is not well discussed with important previous works. I believe that the comparisons and discussions with these works can greatly improve the paper.

I also have some questions about the experiments:
- Is there any reasoning why we have to simplify the concatenation into an addition in Section 3.2? They are not equivalent.

- When generating the fooling images of VGG-att, is the attention module involved, or do you use the same fooling images for both VGG and VGG-att?

Minor comments:
- Fig. 1 -> Fig. 2 in Section 3.1. If not, Fig. 2 is never referred.

References
[A] Huijuan Xu and Kate Saenko. Ask, attend and answer: Exploring question-guided spatial attention for visual question answering. In ECCV, 2016.
[B] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In CVPR, 2016.
[C] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Deep compositional question answering with neural module networks. In CVPR, 2016.
[D] Paul Hongsuck Seo, Zhe Lin, Scott Cohen, Xiaohui Shen, and Bohyung Han. Hierarchical attention networks. arXiv preprint arXiv:1606.02393, 2016.
[E] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In CVPR, 2016.
[F] Jonghwan Mun, Minsu Cho, and Bohyung Han. Text-Guided Attention Model for Image Captioning. AAAI, 2017.
[G] Seunghoon Hong, Junhyuk Oh, Honglak Lee and Bohyung Han, Learning Transferrable Knowledge for Semantic Segmentation with Deep Convolutional Neural Network, In CVPR, 2016.
[H] Max Jaderberg, Karen Simonyan, Andrew Zisserman, Koray Kavukcuoglu, Spatial Transformer Networks, NIPS, 2015



review_title: Has interesting idea but needs improvement

Meta-review: quality: interesting idea to train an end-to-end attention together with CNNs and solid experiments to justify the benefits of using such attentions.
clarity: the presentation has been updated according to review comments and improved a lot
significance: highly relevant topic, good improvements over other methods
Decision: Accept (Poster)