Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a Traversal Network that traverse the image scale-space in a top-down fashion, selecting only the most promising areas to compute features on.

The paper propose learning a simple policy for hard-attention to save computation resources in high-resolution image classification.

main_review: # Originality:
Minor originality, the paper builds on previous work about hard-attention and combine well know RL algorithm REINFORCE to learn a simple policy for attention.

# Quality: 
Medium quality, there are multiple baseline missing, especially DNN which are designed to be more efficient, ex. EfficientNet, and which are competitive with the proposed solution in terms of FLOPs and accuracy. It's unclear if the proposed method could be applied to more efficient networks.

There is no comparison with random or fixed locations (ex. center crop in multiple resolutions) as reasonable baselines.


# Clarity: 
Overall, the paper is clear and well written. 


# Significance:
The significance of the work is low, there is not comparison with strong baselines of architectures designed for efficiency or high-resolution images.

Furthermore, the paper doesn't explore other relaxation methods to propagate gradients via sampling, ex. "Categorical Reparameterization with Gumbel-Softmax".

# UPDATE:
Based on the authors detailed comments and other reviews, I've have updated my score to 6.

limitations_and_societal_impact: The authors only briefly hypothesize the the broader impact of their approach, including the possibility of reduce of resource utilization. However, authors didn't mention any potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes TNet (Traversal Network), a hard attention architecture for classification of high-resolution images using a pyramid of different resolution versions of the input. The proposed method:
- applies a "Feature extraction module" to a 224 x 224 image
- applies a "Location module" to the spatial output of the previous module to select which sub-regions of the input should be further processed
- recursively applies the same steps to each selected subregion, but at a higher-resolution

Each time the "Feature extraction module" is run, it generates a fixed length feature embedding. These embeddings are passed through a "Positional encoding module" and the resulting embeddings are averaged to create a final fixed length representation of the input. The fixed representation can finally be used in a logistic regression classifier.

As this model must dynamically choose which parts of the image are used in the next iteration, it is not end-to-end differentiable, and thus uses a REINFORCE based learning process with M sampled decisions per image to approximate the loss.

The authors test this method against baseline methods on ImageNet, fMoW, CUB-200-2011, and NAbirds. The baseline methods include Saccader, DRAM, BagNet, and EfficientNet. 
main_review: This paper's contributions are in 1.) a new hard-attention method that can be applied recursively to high-resolution images, and  2.) benchmarking experiments to show the effectiveness of this method against previous work.

The first contribution is novel and interesting, however the experiments for the second contribution are not convincing. For example, the set of experiments on ImageNet do not show the benefits of the proposed approach. The standard procedure for reporting performance on ILSVRC2012 involves resizing images to 224x224 which is not high-resolution, and there exist baseline models with much higher performance and throughput than BagNet-77 (like the EfficientNets). The authors should consider a scatter plot to show model throughput vs. top-1 accuracy to illustrate the pareto frontier of _all_ types of classification models in addition to Table 1. Currently, it is impossible to evaluate how the proposed method compares to other work on this tradeoff.

The argument for efficient performance in the high-resolution setting is more sustained with the fMoW dataset, as satellite imagery (and medical imagery) can be magnitudes larger than "natural" images. However in this case the authors do not compare to SOTA methods. For example, [1] shows top-1 accuracy results >75%.

The results suggesting that Location modules intelligently select subregions of the imagery are encouraging and should be further explored in the remote sensing or medical imaging contexts where the objects of interest _aren't_ in the center crop of the input.


[1] Minetto, Rodrigo, Maurício Pamplona Segundo, and Sudeep Sarkar. "Hydra: An ensemble of convolutional neural networks for geospatial land classification." IEEE Transactions on Geoscience and Remote Sensing 57, no. 9 (2019): 6530-6541.


Minor points
- How many samples, M, are needed to learn / used? Is this an important hyperparameter?
- 140-142: b and R are not used until equation 2 in the next section and should be described there instead.
limitations_and_societal_impact: -
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper reduces the computational cost of processing large images by leveraging a multi-scale hard attention mechanism. Starting from a low resolution version of an image, it progressively identifies and zooms in the most discriminative areas of the image. By ignoring most of the irrelevant content of the image, it processes images faster than strong traditional approaches, while achieving higher accuracy than these baselines thanks to a novel regularization mechanism.
main_review: The paper makes 3 contributions:
 * It combines a multi scale approach, such as the one successfully used for StreetView house numbers, with a hard attention mechanism. This is a clever approach and a significant contribution.
 * It introduces a new per-feature regularization mechanism that ensures that non discriminative glimpses are identified and penalized in the loss. This is a very ingenuous idea, and like many brilliant ideas, it is both simple and appears obvious in retrospect (yet nobody has thought of it until now as far as I know).
* The model leverages a novel neural network architecture built around 4 modules working together. The whole architecture is flexible, and can be rebuilt as needed around various convolutional models called backbones in the paper (e.g. Bagnet-77,  EfficientNet). This is solid work.

The combined solution is validated by an extensive empirical evaluation. The most significant results demonstrate that:
 * it very significantly outperforms previous hard attention based approaches (namely DRAM and Saccader) both in terms of FLOPs and accuracy
 * it slightly outperforms the backbone model on which it's build (Bagnet-77) in terms of accuracy but significantly outperforms it in terms of FLOPs.
 * it outperforms Resnet50 and Densenet-151, two strong baselines both in terms of FLOPs and accuracy.

The paper also provides numerous examples of images and the corresponding glimpses that were identified by the policy. A visual inspection of these examples provides convincing evidence that the policy is focusing on what we, as human, would consider the most important parts of each image. However, I would have preferred a less anecdotal approach. Perhaps the authors could, for example, evaluate how masking the glimpses identified by the policy impacts the accuracy of the prediction, and compares this to the impact of masking random parts of the image. 

Last but not least, the paper attempts to quantify the impact of their per-feature regularization mechanism. Their analysis demonstrates that this technique is a key element to the improved performance compared to previous hard attention work. 

The paper is very well written. It is well organized, which makes it a breeze to follow. It was a pleasure to read. Furthermore, the approach can either deliver increased accuracy given a fixed compute budget, or reduce the compute cost given a fixed accuracy target. This tradeoff can be controlled simply by dialing the number location attended to. As a result, and thanks to its impressive results, I believe the paper will get a lot of attention (pun intended)

limitations_and_societal_impact: The authors mentioned the main limitation of the work, which is that the number of glimpses to attend to needs to be tuned manually to achieve a desired level of accuracy, and needs to be retuned for each dataset to achieve optimal performance. They also mention a number of follow up directions to further strengthen the work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors present a hard attention mechanism, called  TNet,  for hi-res image classification.  The methods employ a resolution cascade, where a feature network attends to only a few number of locations of each resolution. The non-differentiable attention sampling is optimized with a Reinfore-like learning rule. All feature vectors are simply pooled for final classification, thus avoiding overhead from recurrent processing.
The authors compare their algorithm with two other hard-attention methods, Saccader and DRAM, and one fully convolutional architecture, BagNet-77 on ILSVRC 2012 ImageNet. Results on the opening figure indicate  that TNet is superior to the other methods on the Pareto frontier of accuracy and GFLOPS per image.
main_review: ## Originality
The idea of doing hard attention is not new, and as such, the TNet architecture does not come with too many big surprises. Please note, that this is not a criticism. What makes the work original and distinctive is how the authors designed the architecture to benefit from multi-resolution hard (and thus sparse) attention, and, at the same time, maintain computational efficiency. That efficiency is achieved imho, by (partially, on every resolution) parallel processing of feature extraction, and by a not overly complicated attention sampling mechanism. Also the feature aggregation with simple averaging is certainly due to  computational efficiency rather than model expressiveness. 

## Quality
The approach is technically sound. The derivations of the learning rules follow the line of Reinforce. New and surprising to me was to replace $p(y_i|l^i,x_i,w)$ by an indicator function (line 140-141). Although this sounds plausible, I'd loved to get some motivation for it (or a reference).

What's missing is an exploration of the edge cases of the hyper parameters $\lambda_f$ and $\lambda_r,\lambda_c$ (except for $\lambda_r=\lambda_c=1$ in Sec. 5.5).  In your experiments $\lambda_f=0.1$ is not big, so it could be, in principle, that the second term of the learning rules doesn't actually matter. Same for $\lambda_f,\lambda_r$, could be that $L_F^s$ (2a) is actually irrelevant.    
Showing that there is a sweet spot $0<\lambda_{\\{f,r,c\\}}<1$ would remove such concerns. 

## Clarity
The paper is well written and easy to follow. I found it extremely fast to get the key message, just from looking at Figs. 1 and 2.
The authors might consider moving some material from the supplementary to the main paper, in order to make the main paper self contained:
* line 140: What is the baseline $b$. Please put supplementary (5) to the main paper.
* Sec. 5.4, Fig. 4: In order to judge the results, it is necessary to know the *feature weighting module*, which is only explained in the appendix.
*  What is the value of $M$? This linearly affects the GFLOPS/image, no? In the appendix (Sec. 1.2) I've found that actually no sampling is done. This should be elaborate in the main paper, as it may affect the correctness of the learning rule.  

For all results, please indicate whether the GFLOPS are with or without backward processing. Event better, provide both numbers.

## Significance
I think the work is potentially significant, as hires image processing is a real application. The authors could have added to the significance by addressing a really large scale problem (896x896 is not exactly gigantic). Also, the paper could be stronger by not just comparing with other hard-attention models, but also any other efficient image classification approaches. 
If TNet would be still superior, then it would give a clear signal to practitioners to focus on this method. 



limitations_and_societal_impact: The authors address limitations of their model in Sec. 6. I would add that a further limitation is that, by construction, the model can only attend hi-res regions that have been attended to in low-res too (I'm not sure if that's meant with *"scale-space traversal to be bi-directional"*). 

I'm okay with their perspective of the broader impact being in increased resource efficiency, given that in a broader study TNet still can push forward the Pareto frontier of Fig. 1b.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: High-resolution image classification is a hard problem for DNN due to resource constraints. The authors propose a multi-scale hard-attention architecture, TNet, for high-resolution information, by only visiting the most informative image regions, to achieve better accuracy vs complexity tradeoff. The architecture achieves a better tradeoff than other baselines on both ImageNet and fMoW datasets.
main_review: The authors propose TNet, which leverages the high-resolution information with low computation complexity, using a multi-scale hard-attention architecture.  The paper is clearly written, providing good motivation for the proposed method.

**Strengths**
1. The TNet offers a better trade-off between accuracy and complexity, with hard attention and only using part of the information in the image by attending to the limited number of image regions.
2. The method is orthogonal to the backbone model and can be applied to different models.
3. According to experiment results, the model provides a better trade-off between accuracy and FLOPs, as well as the inference time comparing to the SOTA.
4. The design of selecting important regions with attention probability hierarchically utilizing multi-scale information of the high-resolution image.
5. The reinforcement rule introduced to solve the not differentiable problem by selecting locations is interesting and novel.
 
**Weaknesses**
1. The experiments in Figure 3 do not indicate improvement of the TNet against the EfficientNet, especially for B2 version. It is a bit contradict with the significant improvement in Table 2. Also, the accuracy of the TNet for very small FLOPs (0.39B) is extremely bad on the fMoW comparing to the baselines in Table 2.
2. In the ablation study, the authors mention that the per-feature regularization may have a similar impact as cropping-based data augmentation. Though they clarify the difference between the two methods, it would be great to see an empirical comparison of random cropping and the per-feature regularization.

Typos
1. 'Our goal it to' in line 26.
limitations_and_societal_impact: The method proposed could improve the efficiency of the image classification task, and provides the researchers a method to utilize the information in high-resolution images unequally. However, the accuracy of the model under extreme computation limitation is not as good as other baseline models as mentioned above, which constraints the application of the method on limited resources.
review_title: null

Meta-review: This work proposes reducing the computational cost of running inference on high resolution images by leveraging a multi-scale attention architecture. The method progressively employs higher resolution imagery to zoom in regions of interest that aid in discriminative training, while ignoring any computation on non-informative regions. The resulting model is tested on ImageNet, fMoW and CUB-200-2011 against several baseline architectures, including  Saccader, DRAM, BagNet, and EfficientNet and shows favorable performance in terms of computational cost versus accuracy. The reviewers did voice some concerns about the lack of testing on a high resolution imagery, which remain unresolved. Overall, the reviewers were favorable with this work, and the concerns were notable, and did not rise to a significant enough level. This paper will be accepted to the conference.

Decision: Accept (Poster)