Hard-Attention for Scalable Image Classification
Authors: Athanasios Papadopoulos
Pawel Korus
Nasir Memon
Venue: NeurIPS 2021 Poster (2021)
Keywords: classification
hard-attention
multi-scale
scalability
high-resolution
interpretability

Abstract: Can we leverage high-resolution information without the unsustainable quadratic complexity to input scale? We propose Traversal Network (TNet), a novel multi-scale hard-attention architecture, which traverses image scale-space in a top-down fashion, visiting only the most informative image regions along the way. TNet offers an adjustable trade-off between accuracy and complexity, by changing the number of attended image locations. We compare our model against hard-attention baselines on ImageNet, achieving higher accuracy with less resources (FLOPs, processing time and memory). We further test our model on fMoW dataset, where we process satellite images of size up to $896 \times 896$ px, getting up to $2.5$x faster processing compared to baselines operating on the same resolution, while achieving higher accuracy as well. TNet is modular, meaning that most classification models could be adopted as its backbone for feature extraction, making the reported performance gains orthogonal to benefits offered by existing optimized deep models. Finally, hard-attention guarantees a degree of interpretability to our model's predictions, without any extra cost beyond inference.

Hard-Attention for Scalable Image Classification
Athanasios Papadopoulos' Pawel Korus" 1.2 Nasir Memon'
'Tandon School Engineering, New York University 2 FAGH University of Science and Technology (tpapadop, pkorus, hemon)@nyu. edu
Abstract
Can we leverage high-resolution information without the unsustainable quadratic complexity imput scale? We propose Traversal Network (TNet), a novel multi- scale hard-attention architecture, which traverses image scale-space in a top-down fashion, visiting only the most informative image regions along the way. TNet offers an adjustable trade-off between accuracy and complexity, by changing the number of attended image locations. We compare our model against hard-attention baselines on ImageNet, achieving higher accuracy with less resources (FLOPs, pro- cessing time and memory). We further test our model on fMoW dataset, where we process satellite images of size up 896 x 896 px. getting up to 2.5x faster process- ing compared to baselines operating on the same resolution, while achieving higher accuracy well. TNet is modular, meaning that most classification models could adopted as its backbone for feature extraction, making the reported performance gains orthogonal to benefits offered by existing optimized deep models. Finally, hard-attenn attention guarantees a degree of interpretability to our model's predictions, without any extra cost beyond inference.
Introduction
In image classification, deep neural networks (DNNs) are typically designed and optimized for specific input resolution, e.g. 224 x 224 px. Using modern DNNs images of higher resolution (as happens c.g., satellite or medical imaging) is a non-trivial problem due to the subtlety of scaling model architectures [57]. and rapid increase in computational and memory requirements.
linear increase the spatial dimensions of the input, results in quadratic increase in computational complexity and memory, and can easily lead to resource bottlenecks. This can be mitigated with careful engineering, e.g., streaming [46] gradient checkpointing
42 However, such solutions are content-agnostic, and don't 't take advantage of the fact that discriminative information may be sparse and distributed across various image scales, deeming processing of the whole input unnecessary.
Our goal is to leverage high-resolution information, while dispensing with the unsustainable quadratic complexity input scale. To this end, we propose Traversal Network (TNet), a multi-scale hard- attention architecture, which traverses image scale-space in top-down fashion, visiting only the most informative image regions along the way. TNet is recursive, and can be applied inputs of virtually any resolution; an outline of its processing flow is presented in Fig. (a). Our method draws its intuition from the way humans use saccades to explore the visual world.
TNet offers an adjustable trade-off between accuracy and complexity, by changing the number of attended image regions. This way, complexity increases linearly with the number of attended locations, irrespective of the input resolution. Also, hard-attention explicitly reveals the image regions that our model values the most, providing certain degree of interpretability (Fig. (c)). Importantly, interpretability comes without any extra cost beyond inference, in contrast to popular attribution methods, which require least additional backward pass (53). or numerous forward passes (I]
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

7h
701
65
60
05
BagNe-277
Stoester
50 0
TNc.
2.5 5.0
12.5 GPLOPs
1500
20.0
(a)
(b)
(c)
Figure I: (a) Multi-scale processing in TNet. Starting at level features selectively extracted from image regions at various scales (red cubes), and then, they are combined to create the final image representation used for classification (blue cube). (b) Experimental results on ImageNet [I3 with baselines based on [8]. Numeric annotations correspond to the number attended locations. Our model offers better trade-off between accuracy and complexity (FLOPs). (c) Examples of attention policy (top 3 locations) learned on ImageNet.
Attention may also reduce data acquisition cost 660 by allowing only fraction of the high-resolution content to acquired.
TNeet is trained end-to-end by employing a modified version of REINFORCE rule [68 while using only classification labels. Our architecture is modular, and most classification models could be adopted as its backbone for feature extraction. This way, we can directly take advantage of various performance benefits offered by existing optimized deep models.
Hard-attention the mechanism that allows TNet to dispense with quadratic complexity to input scale, and as result, evaluate model against strong hard-attention baselines ImageNet (18). summary of our results depicted Fig. (b), where we see that TNet offers a better trade-off between accuracy and complexity measured FLOPs (similar behavior is observed with actual timings and memory). We extend our experiments to fMoW dataset, which consists of high-resolution satellite images [12). We process images up 896 896 px, getting up 2.5x faster processing compared to baselines operating on the same resolution, while achieving higher accuracy as well.
We find improvements in accuracy surprising, because TNet is processing only part of the input, in contrast to fully convolutional baselines. We primarily attribute this behavior to a novel regularization method, which encourages classification based individual attended locations. We verify its efficacy through an ablation study.
Related work
Attention. Attention has a long history in the artificial neural networks literature 31 and in the
modern era of deep learning it has been used very successfully in various problems 36 14 4 770 Two main forms attention are: soft attention which processes everything but weights various regions differently: and hard atlention which selects only a fraction of the data for processing. Hard-attention models address various use-cases, and can be motivated by interpretability [18], reduction of high-resolution data acquisition cost [60], computational efficiency to offer single model that takes advantage all these benefits.
22 67 (5)
33 Our goal is
Our model is conceptually similar to glimpse-basee based models 2 difference is that don't restrict our attention policy to 2D space, but we consider scale dimension as well. Also, our model parallelizes feature extraction at each processing level, instead of being fully sequential. Furthermore, we don't 1t use recurrent neural networks (RNNs) to combine features from different locations, but instead, we simply average them (see Section 3 This way, gradients flow directly to extracted features, without the need to backpropagate through RNN steps. This is simpler strategy compared to LSTM in order to avoid vanishing gradients.
50
19
An important

---

T Input imzage pr)
LVL [E
Tresl
Phellonel
Vodul
Nodul
Class Probabiliies
Candidare locations
W Sampling
Vodule
V21
LVV
Ecrrction Modile
Encoding
Madele
Nodule
445 445
Candidate locations
V23
V23
Sampling
Vodile lo
V35
Feot
Encading Nedalo
Vodil de
LVL
V39
Figure 2: Three unrolled processing levels of our architecture. Starting at level 1, the image is processed in the coarsest scale (Feature Extraction Module), and the extracted features are used to decide which image locations should be processed in finer detail (Location Module). This process is repeated for each selected location to reach level 3, where features from the highest resolution extracted. All features are enriched with positional information (Positional Encoding Module), and then averaged before the final classification (Classification Module).
Recent work explores soft attention mechanisms based on transformers, which originate from the natural language processing community [63] Transformers have already been used extensively in machine vision 45 74
16
g 48
and research interest related directions has increased [6] 11 5
Multi-scale representations. We identify four broad categories of multi-scale processing methods.
(1) Image pyramid methods extract multi-scale features by processing multi-scale inputs [17 47 33
44]. Our model belongs to this category, and due its recursive nature, can extract features from an arbitrary number of pyramid levels (see Section 3p. (2) Encoding schemes take advantage of the inherently hierarchical nature of deep neural nets, and reuse features from different layers, since they Decoding schemes follow up the forwafd processing (encoding) with a decoder, that gradually recovers the spatial resolution
contain information of different scale (24 10] (3) Encoding-Decodingg
of early feature maps, by combining coarse and fine features 152 40 (4) Spatial modules
incorporated into the forward pass, to alter feature extraction between layers 71 9 65
Computational efficiency. There are multiple ways to adjust the computational cost of deep neu- ral networks. We organize them into four categories. (1) Compression methods aim to remove
redundancy from already trained models [38 26 72). (2) Lightweight design strategies
used
to replace network components with computationally lighter counterparts [32] 30 511 27 66). (3) Partial computation methods selectively utilize parts of network, creating paths of computation with different costs (37, 20 73 29 [69] (4) Astention methods selectively process parts the input,
based their importance for the task at hand 49 39 33 56). This is the strategy we follow in
architecture.
Architecture
3.1 Processing flow
We present our architecture by walking through the example in Figure 2 where we process an image with original resolution of 896 x 896 px (1) in the top left corner). In the first level, we downscale the image to 224 x 224 px and pass it through the feature extraction module, order to produce a feature vector Vi that contains a coarse description of the original image.
To proceed to the next level, we feed an intermediate feature map, Fi, from the feature extraction module to the location module, which considers number of candidate locations described by Fi,

---

and predicts their importance (in this particular example, the candidate locations form a 2 x 2 regular grid (2), and the location module yields 4 predictions). We express region importance as attendance probability, which parametrizes categorical distribution used for sampling without replacement; in our current example, we sample" locations (3).
In the 2nd processing level, we crop the selected regions from the full-resolution image, resize them 224 224 px, and feed them to the feature extraction module to obtain the corresponding feature vectors (here 1/2 and V2a). The original input resolution allows us to move to a 3rd processing level, where we feed F21 and Fas to the location module, leading 2 Categorical distributions. We sample 1 location from each one of them to get V35 and V39-
Features extracted from all levels are passed through the positional encoding module, which injects information about the spatial position and scale of the image regions the features describe. The resulting vectors, (V"). are averaged (4) into a single comprehensive representation, V', that is fed the classification module for the final prediction.
3.2
Modules
The feature extraction module receives an image of fixed size as input, and outputs a feature vector V and intermediate spatial representation F. The input size, hyperparameter call base resolution, defines the minimum amount of information that can be processed. Hence, constrains the minimum cost that our model has pay in computational and memory resources. In our experiments, feature extraction modules are implemented using CNNs.
The location module predicts K = n probabilities of Categorical distribution over the locations within a given n X n grid of candidate image regions. It receives a feature map of size n X n e as input, where each x 1 X vector describes the corresponding location of the grid. The feature map is passed through a series squeeze and excitation (28) yielding K logits, which transformed to relative region importance via a softmax layer.
X 1 convolutions (contextual information is infused as well, c.g., via
The positional encoding module receives a feature vector f and positional encoding vector p. and combines them (e.g., through a fully connected layer) to output feature vector S We use variant of the fixed positional encodings based on sine and cosine functions introduced in 633 Instead of single dimension of time, we have three: two spatial dimensions and scale.
The classification module projects the final feature vector (c.g., via linear layer) to classification logits. We provide the exact module architectures in Appendix 2.1, along with justification of our design choices.
Training
4.1 Learning rule
Our model not end-to-end differentiable because of location sampling. We address this problem using a variant of the REINFORCE (68] learning rule:
2 Ologp(u,l'' Ni, w) A b)
2 Plogp(1' p(l (ra, w)
LF N M Σ
ow
Ow
where w; the i-th image, M1 is its label, and w are the parameters of our model. p(1|a [2t, w) is the probability that the sequence of locations is attended for image I,, and p(y. my w) is the probability of predicting the correct label after attending to l'.
N M is the total number of examples used for each update. The size of our original batch B is N, and we derive using a Monte Carlo estimator with M samples to approximate the expectation
En p(1 [r,, w)
Oog Olog p(y.]l'.....)) + log pl (Bsl"" 24, w) poogp(("' p(..e((( for each image I; in B. To 8w
8w
reduce the variance of the estimator, we replace log Dgg)(u:)? 24, w) with a discrete indicator function R, which is equal to for correct predictions and 0 otherwise (2). To the same end, we use baseline b, which corresponds to the exponential moving average of the mean reward R, Vi, and is updated

---

after processing each training batch [70]
NM E R
bn 0.9 -n- 0.1
NM
where R. is the reward for the i-th image in the n-th batch. For simplicity. drop the subscript of b, in (T. y is a weighting hyperparameter. We provide a detailed derivation of our learning rule in Appendix 1.1.
The first term of T.p used to update the parameters in order to maximize the probability of the correct label. The second term is used to update the location selection process, according to the utility of the attended sequence 1 in the prediction of the correct label.
4.2 Per-feature regularization
When our model attends to location sequence I; and makes a correct prediction, we positively reinforce the probability attend every location in the sequence. This is expressed in the second term of I where we use probability p(t w) of the whole sequence 4, However, some locations may not contribute to the correct prediction, e.g., if they have missed the object interest. In such cases, we reinforce the attendance of uninformative regions, encouraging a sub-optimal policy.
To mitigate this problem, for every attended location, classification prediction. Then, we use these predictions to complement our assessment on whether the corresponding image regions have useful content. We modify our learning rule as follows:
its feature vector make separate
L, Lا
+
N-M
A a log p(yill Ti, w)
log Dog la,, w)
1.p
+ AR -b)
(3a)
N M
Ow
Ow
N-M
2 log p(yn Hz, w)
2 log plp (2 w)
13 N M
(1 Xa)
+ Ar (.) b)
(3b)
Ow
Dw
where Li is learning rule ①T) with additional weighting hyperparameters An, [0, 1]. learning rule [ when we attend only to the k-th location, G from every sequence (t, Also, introduce weighting factors (1 Ae) and (1 - Ar). During training, we attend to a fixed number of [P'] locations for every image. R₁ and R. are discrete indicator functions equal to when a prediction is correct, and 0 otherwise. Lip updates the parameters of our model based attending /' Vi, while L updates them based on ; X and A specify the relative importance of these updates.
LE is
Even though our initial motivation was to improve the attention policy, the first term in 3bb updates feature extraction parameters based on independent predictions from attended image regions. We empirically observed that such updates boost performance, potentially because they lead to features that co-adapt less and generalize better.
Experimental evaluation
5.1 Effectiveness of hard-attention mechanism
Data. ImageNet [13
consists of natural images from 1, 000 classes. We use the ILSVRC 2012 version, which consists of 1, 281, 167 training and 50, 000 validation images.
Models. We use Saccader and DRAM [8] as hard-attention baselines. We use Saccader with Bag [7] as its backbone for feature extraction. BagNet-77-lowD is based ResNet-50 [2], with receptive field constrained to 77 x 77 px. DRAM uses the standard ResNet-50 for feature extraction, with glimpses of 77 77 px.
For fair comparison, we set the base resolution of TNet to x 77 px, and use BagNet-77, slightly modified version of BagNet-77-lowD, as the feature extraction module. In the location module,
5

---

Table I: Efficacy of hard-attention mechanism: TNet surpasses all hard-attention baselines [8 on ImageNet, by attending to just location. Higher accuracy can be achieved for less FLOPs, and translates to lower actual run time. Memory savings obtained compared BagNet-77 and Saccader as well. TNet has slightly more parameters compared to BagNet-77 due to the additional modules, but has significantly fewer parameters than the hard-attention baselines. FLOPS, time, and memory are measured during inference.
Top-1
Top-5
FLOPs
#Params (M)
Time (msec/im)
Memory (GB)
Model
#Locs
Acc.
Acc.
(B)
2
67. 97.79%
85.42%
19.5
7.31 +: 1.55
2.58
6
87.8%
19.5
7.32 + 1.55
2.52
Saccader
35.58
20
72.02%
89.51%
19.51
7.36 1.50
2.53
72.27% 30
89. 79%
19.51
7.36 + 1.49
2.51
2
49.72%
73.27%
1.86
3.43 1.57
0.45
4
64.26%
84.4%%
3.1
3.92 + 1.56
0.44
DRAM
45.61
8
67.5%
86.6%
5.58
4.61 + 1.59
0.45
65.15% 20
84.58%
13.03
7.58 + 1.53
0.46
BagNet-77
73.42%
91.1%
18.42
20.55
5.94 =: 0.09
2.62
0
67.29%
87.38%
1.82
0.74 + 0.01
0.46
1
73.12%
90.56%
3.63
1.43 1 0.01
0.57
TNet
21.86
2
74.12%
91.18%
5.43
2.09 1 0.02
0.69
3 74.41%
91.4%
7.24
2.74 0.03
0.95
5
74.62%
91.35%
10.84
3.96 + 0.04
1.47
use uniform 5 x 5 grid of overlapping candidate regions. The dimensions of each grid cell span 34.375% of the corresponding image dimensions. This way, for an image of size 224 x 224 px. the image patches within the grid cells at the 2nd processing level are x 77 px. We use Net-7t as separate fully convolutional baseline, with inputs of 224 x 224 px. Additional details about TNet and BagNet-77 77 are provided in Appendix 2.1.1.
Training, We train TNet with 2 processing levels on images of 224 x 224 using class labels only. We train for 200 epochs using the Adam optimizer 33 with initial learning rate 10 that we drop once by a factor of 0.1. We use dropout (keep probability 0.5) in the last layer of feature extraction. We use fer-fue regularization with A X, 0.3. We attend a fixed number of 3 locations.
We train BagNet-77 from scratch, on 224 x 224 px images. Compared to TNet, we reduce dropout keep probability to 0.375, and we early-stop 175 epochs. We don't 't train own Saccader and DRAM, we use results reported in 18] Additional training details are provided in Appendix 2.2.1. 1.
We present results in Table TNet outperforms Saccader and DRAM by attending to only location, while it surpasses BagNet-77 by attending to 2. Saccader was designed with accuracy and interpretability in mind, which leads to sub-optimal computational efficiency, as it processes the entire full-resolution image before attending to locations. As a result, FLOPs stay nearly constant and remain similar to BagNet-77, more that 5 times higher than TNet with 1 attended location. DRAM offers the expected gradual increase in computational cost as the number of attended locations increases, but for FLOPs comparable with TNet e.g., DRAM for 8 locations (maximum accuracy) and TNet for 2 our model is superior by more than 6.5%.
Results.
TNet has slightly more parameters than BagNet-77, because of the location and positional encod- ing modules. Saccader and DRAM have significantly heavier attention mechanisms in terms of parameters.
We profile all models to validate correspondence between theoretical FLOPs and real processing speed, and to assess memory requirements. We time inference on batches of 64 images; early batches are discarded discount code optimization. We use single NVIDIA Quadro RTX 8000 GPU, with 64 GB of RAM, and 20 CPUs to mitigate data pipeline impact. For Saccader and DRAM we use public implementations 21 in TensorFlow (TF) 1. TNet and BagNet-77 are implemented TF 2. The difference in TF versions may be confounding factor in the obtained results. However, TNet and -7 use the same environment and yield the expected difference in latency. DRAM attends to locations sequentially, while TNet processes all locations from the same level in parallel.

---

Table 2: TNet effectively scales to images of resolution 448 448 px and 896 x 896 px fMoW [12 surpasses accuracy EfficientNet- BO baselines trained on inputs of the same resolution, while requires less FLOPs. Differences in FLOPs translate to differences in actual run time, while memory requirements are lower as well. TNet has more parameters compared to EficientNet-BO due to the additional modules. FL.OPs, time, and memory are measured during inference. A graphical representation of the main results is provided Appendix 2.4.1.
Input Size
Top-1
Top-5
FLOPs
#Params (M)
Time (msec/im)
Memory (GB)
Model
#Locs
BBoxes
Acc.
Acc.
(B)
EfficientNet-BO
221º
69.7%
89.22%
0.39
4.13
0,80 + 0.01
0.76
ResNet-50 [0
224
67.3%
4.09
23.71
DenseNet-121 6
224
70.7%
3
7.1
221a
62.8%
81.97%
0.39
0.80 + 0.01
0.76
EfficientNet-B00
448²
69.83%
90.22%
1.54
4.18
3.04 + 0.18
2.6
896
70.6%
90.81%
6.18
12.18 + 0.18
10.35
221²
47.79%
81.14%
0.39
0.81 0.02
0.76
70.17%
90.99%
0.77
1.74 0.02
1.3
TNet
448²
2
71.46%
91.58%
1.16
4.56
2.46 + 0.03
1.55
3
71.57%
91.72%
1.55
3.18 +: 0.04
2.03
1
72. 16%%
91.98%
1.94
4.71 + 0.03
3.13
896
71.92%
91.83%
2.71
6.13
3.47
This leads to fixed memory use in DRAM and monotonic increase in TNet, as the number of attended locations increases. We could trade this for latency by switching to fully sequential processing.
5.2 Scalability
Data. Functional Map of the World (fMoW) [12 consists of nigh-resolution satellite images from 62 classes. They are split in 363, 572 training, 53, 041 validation and 53, 473 testing images. All images have bounding box annotations.
the feature extraction module of TNet, with base resolution 224 224 px. In the location module we use a x grid with overlapping cells (50% overlap). Hence, when processing 2 levels, we use inputs of 448 x 448 px, and with 3 levels, we use inputs 896 896 px. Details of the TNet architecture are provided in Appendix 2.1.2.
Models. We use EfficientNet-Bo (EN-BO) 57
We create our first baselines by training EN-BO with images resized to different resolution; 224 x 224 px, 448 448 px, and 896 896 px. We use the available bounding boxes crop the regions of interest, resize them to 224 x 224 px, and train an additional EN-BO bascline. Also, we report the best accuracy we find in the literature, achieved by single-model predictions (not network ensembles), and without the use of any meta-data for training. These are the ResNet-50 60 and DenseNet-121 [6] baselines.
Training. We train TNet using only classification labels. The training proceeds stages: (1) training from scratch for 40 epochs on 448 448 px inputs, using processing levels, and fixed number of 2 locations; (2) fine-tuning for 10 more epochs on images of 896 x 896 px, which allow us extend processing
levels. We 4 locations; 2 in the 2nd processing level, plus 1 for each of them in the 3rd. We use per-feature regularization in both stages. All EN-BO baselines are trained independently. from scratch. The exact hyperparameters are provided in Appendix 2.2.2.
Results. We present our results in Table 2 TNet surpasses all basclines by attending to 2 locations (2 processing levels), with top-1 accuracy of 71.46%. This requires less FLOPs than the corresponding EN-BO baseline at the same input size (448 x 448 px), which achieves accuracy of 69.83%.
Extending processing to 3 levels with 4 attended locations, further increases accuracy to 72.16% while FLOPs increase by ~ 67% (from 1.16 to 1.94B FLOPs). The corresponding EN-BO baseline operating 896 x 896 px inputs, achieves 70.6% accuracy, and requires 300% more FL.OPs compared to EN-BO operating on 448 448 px inputs (from 1.54 to 18B FLOPSs. This shows that TNet can efficiently leverage high-resolution information.
7

---

CUB 200 2011
NABirds
90
86-
Accuracy 10p-1
96
top-1
TN
TN
TNet B2
80
TNet B2
TNet-B4
TNet-B4
EN-B0
EN-BX DX
76-
EN-BI
74
EN
6
GFLOPs per image
per image
Figure 3: TNet is modular, and different models can be used its feature extraction module. We plot TNet performance with different backbones (attended locations range from 0 to 5). Stronger backbones lead to the expected increase in accuracy and FLOPs. EN baselines and the feature extraction modules of TNet models are initialized with pre-trained weights, while the rest of the modules are randomly initialized. TNet models achieve similar or better accuracy compared to corresponding baselines, showing that fine-tuning can be an effective practice.
We profile our models by following the procedure described in Section 5.1 and show that differences in FLOPs translate to differences in actual run time. TNet also requires less memory, although has parameters because of the location and positional encoding modules.
We note that when TNet limits its processing to just level (it uses only the contextual feature vector, fer extracted from the downscaled version of the whole image), its accuracy is considerably lower compared to EN-BO operating on 224 x 224 px inputs. We hypothesize that as TNet is trained with more processing levels, fc co-adapts with the increasing number of extracted feature vectors. To mitigate this and maintain high accuracy at every processing level, as part of our future work, we update Equation 3b appropriately weighting terms that correspond to feature vectors from different levels.
5.3 Modularity and fine-tuning
TNet is modular architecture, and we would like to test its compatibility with different backbone models, especially when these models are initialized with pre-trained weights, since this a popular practice [15 [6. 58
and NAbirds 62 which are commonly used as downstream tasks. CUB-200-200 has 200 classes of bird species, with 5, 994 training and 5, 794 testing images. NABirds has 555 classes of bird species, with 23, .29 training and 24, 633 testing images. We resize all images to 448 x 448 px.
Data. We use 2 fine-grained classification datasets, CUB-200-211
Models. We use baselines from the EfficientNet (EN) family [57], EN-Bi, E 1....... 4}. We use the same EN models as the feature extraction module of TNet, getting TNet-Bi. All TNet models have base resolution 224 224 px, attention grid of 5 5, and each grid cell dimension occupies 35% of the corresponding image dimension. Processing extends to 2 levels.
a weighted average replace the simple averaging of the extracted feature vectors before the final prediction (4 in Fig. 2 The weights for the averaging are calculated by new feature weighting module, which receives the N feature vectors extracted from all processing levels, and outputs N weights that sum up to 1. Details about all architectures are provided in Appendix 1.3.
For TNet models,
Training. EN baselines and the feature extraction modules of TNet models are initialized with weights pre- trained on ImageNet [59]. The rest of TNet modules randomly initialized. All models are fine-tuned on the downstream tasks. Training details are provided Appendix 2.2.3.
Results. We summarize results in Figure 3 For clarity, we limit the number of models we plot, and we provide detailed results in Appendix 2.4.2. We see that performance differences between baselines, translate to similar differences between TNet models with corresponding backbones,

---

0.05
0.52
0.02
0.6
0.01
D3
0.01
0.04
D
Figure 4: Examples of attention policy learned on CUB-200-2011. Numeric annotations correspond weights predicted by the feature weighting module for the top 3 locations and the downscaled version of the whole image (Ist processing level). Weights sum up to 1.
e.g.; EN-B4 and TNet-B4 achieve considerably better accuracy compared to EN-BO and TNet--O respectively. This indicates that behavioral differences between classification networks can be manifested TNet models well, if these networks are used feature extraction modules.
In addition, as attended locations increase, TNet models achieve similar or better accuracy compared to corresponding baselines [TNet-B4in is also competitive to strong baseline API-Net [75] and state-of- the-art TransFG 23:: see Appendix 2.4.2). This indicates that initialization with pre-trained weights allows TNet to learn useful weights for all modules.
5.4 Attention policy and interpretability
We show examples of attended locations in Fig. I (c) and Fig. more examples from all datasets provided in Appendix 3.2. On ImageNet, the learned policy is highly content dependent and informative, as overlaps with intuitively meaningful regions. The same behavior is observed in CUB-200-2011 and NABirds datasets, where the weights estimated by the feature weighting module have significant contribution to the interpretability of the predictions. On fMoW, our model predominantly attends locations the center, where nearly all objects of interest are located. While this is a correct heuristic implied by the data, it leads to a biased attention policy that is not sensitive to content changes. This demonstrates that sufficient diversity is needed to train fully informative policies.
To examine the relative importance between the coarse context and the information from the attended locations, we evaluate TNet on ImageNet, by using features only from attended locations. With 2, and 5 attended locations, TNet achieves top-1 accuracy 67. 67.95% 69.66% and 71.05% respectively (with contextual information included, the corresponding accuracy values are 74.12%, 74.41% 11% and 74.62%) The Saccader achieves similar accuracy for 2, 4 and 8 locations, with 67.8%, 69.51% and 70.08%. This shows that features extracted from attended locations have discriminative value even without global context. As result, the attention policy can be valuable for interpreting predictions.
Hard-attention can also be used for reduction of high-resolution data acquisition cost [60]. On MMoW, when our model attends to locations in 3 processing levels, it covers 37.91% of the input in resolution of 448 x 448 px, and less than 12.5% in the highest resolution of 896 x 896 px. Further quantitative analysis of the attention policy is provided in Appendix 3.1.
5.5 Ablation study
We examine the effect of per-feature regularization by training TNet on ImageNet without it. With 1 attended location, top-1 accuracy drops from 73. 12% (Table I to 65.21%, while with 3 and 5 locations, accuracy drops from 74.41% and 74.62%, to 67. 67.33% and 68.55% respectively. The drop substantial, placing TNet below both Saccader and BagNet-77 in terms of accuracy.
Per-feature regularization may have similar impact as cropping- based data augmentation, since it forces the model to make independent predictions with features from every attended location. However, the attention policy is not random, but learned, which is crucial for the quality the crops. In addition, we don't get one image crop per epoch, but multiple crops in the same training iteration. We hypothesize that this is important to prevent feature co-adaptation, since the model learns to recognize the same object from multiple crops simultaneously.

---

Conclusion
We proposed a novel multi-scale hard-attention architecture, TNet, that can efficiently scale images of high resolution. By controlling the number of attended locations, TNet can adjust the accuracy- computation trade-off dynamically. We demonstrated the efficacy of our method on ImageNet against strong hard-attention baselines, and we further verified its behavior with high-resolution satellite images (fMoW). The attention policy reveals the image regions deemed more informative by our model, and makes its predictions inherently interpretable.
There are multiple research directions that can address current limitations of our method. First, we would like the decision on the number of attended locations to stem from content-dependent learned policy. In addition, we would like scale-space traversal to be bi-directional, instead of merely top-down, in order for processing to be more adaptive. To the same end, we would like already extracted features to condition the processing of subsequent locations.
On the broader impact of our approach, we hypothesize that under number of assumptions, hard- attention has the potential be useful any intelligent agent that navigates through the immense complexity of the natural visual world. These assumptions are that (1) the available resources limited (2) there are performance constraints e-g. maximum response time, (3) the available information is practically infinite (4) all bits of information are not equally useful, and can even be misleading, e.g. noise. In this context, it would be beneficial for an intelligent agent to prioritize the expenditure of its resources according to the utility of the available information, in order to reach its performance goals; this what a learned hard-attention mechanism can facilitate.
Funding Transparency Statement
The authors did not receive third third-party funding or third-party support for their rescarch work.
References
[1] Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. In International Conference on Learning Representations, 2018.
[2] Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. Multiple object recognition with visual attention. arXiv preprint DrXiv:1412.7755. 2014.
[3] Jimmy Ba, Geoffrey E Hinton, Volodymyr Mnih, Joel Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past. In Advances in Neural Information Processing Systems, pages 4331-4339, 2016
[4] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409. 1e0..44444444 00.3,,473 2014.
[5] Irwan Bello. Lambdanetworks: Modeling long-range interactions without attention. arXiv preprint arXiv:2102.08602, 2021.
[6] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe V Le. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3286-3295, 2019.
[7] Wieland Brendel and Matthias Bethge. Approximating cnns with bag-of-local-features models works surprisingly well on imagenet. arXiv preprint arXiv: 007600. 2019.
[8] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213-229. Springer, 2020.
[9] Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint a7X.v:1706.05887, 2017.
[10] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEFE transactions on pattern analysis and machine intelligence, 40(4): 834-848, 2018.
10

---

[11] Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, and Jiashi Feng, -etss Double attention networks. arXiv preprint arXiv:1810.11559, 2018.
[12] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In CVPR, 2018.
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Icee, 2009.
[14] Misha Denil, Loris Bazzani, Hugo Larochelle, and Nando de Freitas. Learning where attend with deep architectures for image tracking, Neural computation, 24(8):2151-2184 2012.
[15] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprini arXiv: matii.8100.0555, 2018.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint
arXiv:2010.11929 2020.
[17) David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using a multi-scale deep network. Advances in neural information processing systems, pages 2366 2366-2374, 2374, 2014.
[18] Gamaleldin Elsayed, Simon Korbllith, and Quoe V Le. Saccader: improving accuracy of hard attention models for vision. arXiv preprint arXiv:1908. 07644, 2019.
[19] SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pages 3225-3233 2016.
[20] Michael Figurnov. Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1039-1048, 2017.
[21] google research. Gitllub repository that contains the official Saccader and DRAM implemen- tations, 2019. https:/: master/saccader
[22] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. In International Conference on Machine Learning, pages 1462-1471, 2015.
[23] Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang. Yutong Bai, Changhu Wang, and Alan Yuille. Transfg: A transformer architecture for fine-grained recognition. arXiv preprint arXiv:2103.07970 2021.
[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. In European conference on computer vision. pages 346-361. Springer, 2014.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[26] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini urXivv 2015.
[27] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04580 04861, 2017.
[28] Jie Hu, Li Shen, and Gang Sun. waweeezz-and--xxittateee networks. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018
[29] Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Weinberger. Multi-scale dense networks for resource efficient image classification. arXiv preprint arXiv: 09844, 2017.
11

---

[30] Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv: 1602.07360, 2016.
[31] Laurent Itti, Christof Koch, and Ernst Niebur. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on pattern analysis and machine intelligence, 20(11):
1254-1259, 1998.
[32] Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3666 2014.
[33] Angelos Katharopoulos and François Fleuret. Processing megapixel images with deep attention- sampling models. arXiv preprint (ttf:099000031... 2019.
[34] Tsung- Wei Ke, Michael Maire, and Stella X Yu. Multigrid neural architectures. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6665-6673, 2017.
[35] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[36] Hugo Larochelle and Geoffrey E Hinton. Learning combine foveal glimpses with third-order boltzmann machine. In Advances in neural information processing systems, pages 1243-1251, 2010.
[37] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07666 2016.
[38] Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pages 598-605, 1990.
[39] Hila Levi and Shimon Ullman. Efficient coarse-to-fing non-local module for the detection of small objects. arXiv preprint arXiv: ///.////1111115 2018.
[40] Tsung-Yr Yi Lin, Piotr Dollár, Ross B Girshick, Kaiming He, Bharath Hariharan, and Serge J Belongie. Feature pyramid networks for object detection. In CVPR, volume 1, page 4, 2017.
[41] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference computer vision, pages 21-37. Springer, 2016.
[42] Francesco Marra, Diego Gragnaniello, Luisa Verdoliva, and Giovanni Poggi. A full-image full-resolution -rddaneeed-r cnn framework for image forgery detection. IEEE Access, 11.3348--135500 2020.
[43] Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In Advances in neural information processing systems, pages 2204-2212, 2014.
[44] Mahyar Najibi, Bharat Singh, and Larry S Davis. Autofocus: Efficient multi-scale inference. arXiv preprint arXiv: 011..//2.1060 16000 2018.
[45] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz. Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference Machine Learning, pages PMLR, 2018
[46] Hans Pinckaers, Bram van Ginneken, and Geert Litjens. Streaming convolutional neural net- works for end-to-end learning with multi-megapixel images. arXiv preprint arXiv: 1911::9911.444. 2019.
[47] Pedro HO Pinheiro and Ronan Collobert. Recurrent convolutional neural networks for scene labeling. In 31st International Conference on Machine Learning (ICML), number EPFL-CONF. 199822, 2014.
[48] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention vision models. arXiv preprint arXiv:1906.055 05900, 2019.
[49] Jason Ramapuram, Maurits Diephuis, Russ Webb, and Alexandros Kalousis. Variational saccading: Efficient inference for large resolution images. arXiv preprint 1812.333, 2018.
[50] Marc'Aurelio Ranzato. On learning where to look. arXiv preprint
2014.
12

---

[51] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision, pages 225-542. Springer, 2016.
[52] Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer, 2015.
[53] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[54] Pierre Sermanet, Andrea Frome, and Esteban Real. Attention for fine-grained categorization. arXiv preprint arXi:: 1412.7044, 2014.
[55] Yiqiu Shen, Nan Wu, Jason Phang, Jungkyu Park, Kangning Liu, Sudarshini Tyagi, Laura Heacock, S Gene Kim, Linda Moy, Kyunghyun Cho, et al. An interpretable classifier for high- resolution breast cancer screening images utilizing weakly supervised localization. Medical Image Analysis, page 101908, 2020.
[56] Yiqiu Shen, Nan Wu, Jason Phang, Jungkyu Park, Kangning Liu, Sudarshini Tyagi, Laura Heacock, S Gene Kim, Linda Moy, Kyunghyun Cho, et al. An interpretable classifier for high- resolution breast cancer screening images utilizing weakly supervised localization. Medical image analysis, 68: 101908, 2021.
[57] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105-6114, PMLR, 2019.
[58] Mingxing Tan and Quoc V Le. Efficientnetv2: Smaller models and faster training. arXiv preprint arXiv:2104.00298, 2021.
[59] tensorflow. GirHub repository that contains the official EfficientnNer implementation.
2021. URL. httpps:///:/ cwm/ wwom//ensorfll///pprpr http://///.eeeemmmtttt///d//l/// efficientnet.
[60] Burak Uzkent and Stefano Ermon. Learning when and where to zoom with deep reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12345-12354, 2020.
[61] Burak Uzkent, Evan Sheehan, Chenlin Meng, Zhongyi Tang, Marshall Burke, David Lobell, and Stefano Ermon. Learning to interpret satellite images in global scale using wikipedia. arXiv preprint 190Xvv1905.00m 02066. 2019.
[62] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine- one-grained dataset collection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 595-604, 2015.
[63] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint arXiv: I 706.03376. 2017.
[64] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd 20022-200-2011 dataset. 2011.
[65] Huiyu Wang, Aniruddha Kembhavi, Ali Farhadi, Alan L Yuille, and Mohammad Rastegari. Elastic: Improving cnns with dynamic scaling policies. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2258-2267) 2019.
[66] Min Wang, Baoyuan Liu, and Hassan Foroosh. Factorized convolutional neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 545-553, 2017.
[67] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794-7803, 2018
[68] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):229-256 1992.
13

---

[69] Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8817-8826, 2018.
[70] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2057 2015.
[71] Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arxiv 1.07122, 2015.
[72] Ruichi Yu, Ang Li,, Chun-Fu Chen, Jui-Hsin Lai, Vlad Morariu, Xintong Han, Mingfei Gao, Ching- Yung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9194-9203, 2018.
[73] Amir R Zamir, Tc-Lin Wu, Lin Sun, William B Shen, Bertram E Shi, Jitendra Malik, and Silvio Savarese. Feedback networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1308-1317, 2017.
[74] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10076-10085, 2020.
[75] Peiqin Zhuang, Yali Wang, and Yu Qiao. Learning attentive pairwise interaction for fine-grained classification. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 13130-13137, 2020.
14


Reviews:
[{"review_id": "xFcKb_VdpL-", "reviewer": "Reviewer_YvBo", "summary": "High-resolution image classification is a hard problem for DNN due to resource constraints. The authors propose a multi-scale hard-attention architecture, TNet, for high-resolution information, by only visiting the most informative image regions, to achieve better accuracy vs complexity tradeoff. The architecture achieves a better tradeoff than other baselines on both ImageNet and fMoW datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose TNet, which leverages the high-resolution information with low computation complexity, using a multi-scale hard-attention architecture.  The paper is clearly written, providing good motivation for the proposed method.\n\n**Strengths**\n1. The TNet offers a better trade-off between accuracy and complexity, with hard attention and only using part of the information in the image by attending to the limited number of image regions.\n2. The method is orthogonal to the backbone model and can be applied to different models.\n3. According to experiment results, the model provides a better trade-off between accuracy and FLOPs, as well as the inference time comparing to the SOTA.\n4. The design of selecting important regions with attention probability hierarchically utilizing multi-scale information of the high-resolution image.\n5. The reinforcement rule introduced to solve the not differentiable problem by selecting locations is interesting and novel.\n \n**Weaknesses**\n1. The experiments in Figure 3 do not indicate improvement of the TNet against the EfficientNet, especially for B2 version. It is a bit contradict with the significant improvement in Table 2. Also, the accuracy of the TNet for very small FLOPs (0.39B) is extremely bad on the fMoW comparing to the baselines in Table 2.\n2. In the ablation study, the authors mention that the per-feature regularization may have a similar impact as cropping-based data augmentation. Though they clarify the difference between the two methods, it would be great to see an empirical comparison of random cropping and the per-feature regularization.\n\nTypos\n1. 'Our goal it to' in line 26.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The method proposed could improve the efficiency of the image classification task, and provides the researchers a method to utilize the information in high-resolution images unequally. However, the accuracy of the model under extreme computation limitation is not as good as other baseline models as mentioned above, which constraints the application of the method on limited resources.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "sDnOa4aAOXm", "reviewer": "Reviewer_Mp3T", "summary": "The paper proposes a Traversal Network that traverse the image scale-space in a top-down fashion, selecting only the most promising areas to compute features on.\n\nThe paper propose learning a simple policy for hard-attention to save computation resources in high-resolution image classification.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality:\nMinor originality, the paper builds on previous work about hard-attention and combine well know RL algorithm REINFORCE to learn a simple policy for attention.\n\n# Quality: \nMedium quality, there are multiple baseline missing, especially DNN which are designed to be more efficient, ex. EfficientNet, and which are competitive with the proposed solution in terms of FLOPs and accuracy. It's unclear if the proposed method could be applied to more efficient networks.\n\nThere is no comparison with random or fixed locations (ex. center crop in multiple resolutions) as reasonable baselines.\n\n\n# Clarity: \nOverall, the paper is clear and well written. \n\n\n# Significance:\nThe significance of the work is low, there is not comparison with strong baselines of architectures designed for efficiency or high-resolution images.\n\nFurthermore, the paper doesn't explore other relaxation methods to propagate gradients via sampling, ex. \"Categorical Reparameterization with Gumbel-Softmax\".\n\n# UPDATE:\nBased on the authors detailed comments and other reviews, I've have updated my score to 6.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors only briefly hypothesize the the broader impact of their approach, including the possibility of reduce of resource utilization. However, authors didn't mention any potential negative societal impact of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "The paper doesn't address the possible ethical concerns of working with high-resolution images which could include private information or unexpected details.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "ey1lDGsWwDD", "reviewer": "Reviewer_dSSn", "summary": "This paper proposes TNet (Traversal Network), a hard attention architecture for classification of high-resolution images using a pyramid of different resolution versions of the input. The proposed method:\n- applies a \"Feature extraction module\" to a 224 x 224 image\n- applies a \"Location module\" to the spatial output of the previous module to select which sub-regions of the input should be further processed\n- recursively applies the same steps to each selected subregion, but at a higher-resolution\n\nEach time the \"Feature extraction module\" is run, it generates a fixed length feature embedding. These embeddings are passed through a \"Positional encoding module\" and the resulting embeddings are averaged to create a final fixed length representation of the input. The fixed representation can finally be used in a logistic regression classifier.\n\nAs this model must dynamically choose which parts of the image are used in the next iteration, it is not end-to-end differentiable, and thus uses a REINFORCE based learning process with M sampled decisions per image to approximate the loss.\n\nThe authors test this method against baseline methods on ImageNet, fMoW, CUB-200-2011, and NAbirds. The baseline methods include Saccader, DRAM, BagNet, and EfficientNet. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper's contributions are in 1.) a new hard-attention method that can be applied recursively to high-resolution images, and  2.) benchmarking experiments to show the effectiveness of this method against previous work.\n\nThe first contribution is novel and interesting, however the experiments for the second contribution are not convincing. For example, the set of experiments on ImageNet do not show the benefits of the proposed approach. The standard procedure for reporting performance on ILSVRC2012 involves resizing images to 224x224 which is not high-resolution, and there exist baseline models with much higher performance and throughput than BagNet-77 (like the EfficientNets). The authors should consider a scatter plot to show model throughput vs. top-1 accuracy to illustrate the pareto frontier of _all_ types of classification models in addition to Table 1. Currently, it is impossible to evaluate how the proposed method compares to other work on this tradeoff.\n\nThe argument for efficient performance in the high-resolution setting is more sustained with the fMoW dataset, as satellite imagery (and medical imagery) can be magnitudes larger than \"natural\" images. However in this case the authors do not compare to SOTA methods. For example, [1] shows top-1 accuracy results >75%.\n\nThe results suggesting that Location modules intelligently select subregions of the imagery are encouraging and should be further explored in the remote sensing or medical imaging contexts where the objects of interest _aren't_ in the center crop of the input.\n\n\n[1] Minetto, Rodrigo, Maurício Pamplona Segundo, and Sudeep Sarkar. \"Hydra: An ensemble of convolutional neural networks for geospatial land classification.\" IEEE Transactions on Geoscience and Remote Sensing 57, no. 9 (2019): 6530-6541.\n\n\nMinor points\n- How many samples, M, are needed to learn / used? Is this an important hyperparameter?\n- 140-142: b and R are not used until equation 2 in the next section and should be described there instead.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"ethical_concerns": "-", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "c5vB_lI4Mp-", "reviewer": "Reviewer_BkyT", "summary": "The paper reduces the computational cost of processing large images by leveraging a multi-scale hard attention mechanism. Starting from a low resolution version of an image, it progressively identifies and zooms in the most discriminative areas of the image. By ignoring most of the irrelevant content of the image, it processes images faster than strong traditional approaches, while achieving higher accuracy than these baselines thanks to a novel regularization mechanism.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper makes 3 contributions:\n * It combines a multi scale approach, such as the one successfully used for StreetView house numbers, with a hard attention mechanism. This is a clever approach and a significant contribution.\n * It introduces a new per-feature regularization mechanism that ensures that non discriminative glimpses are identified and penalized in the loss. This is a very ingenuous idea, and like many brilliant ideas, it is both simple and appears obvious in retrospect (yet nobody has thought of it until now as far as I know).\n* The model leverages a novel neural network architecture built around 4 modules working together. The whole architecture is flexible, and can be rebuilt as needed around various convolutional models called backbones in the paper (e.g. Bagnet-77,  EfficientNet). This is solid work.\n\nThe combined solution is validated by an extensive empirical evaluation. The most significant results demonstrate that:\n * it very significantly outperforms previous hard attention based approaches (namely DRAM and Saccader) both in terms of FLOPs and accuracy\n * it slightly outperforms the backbone model on which it's build (Bagnet-77) in terms of accuracy but significantly outperforms it in terms of FLOPs.\n * it outperforms Resnet50 and Densenet-151, two strong baselines both in terms of FLOPs and accuracy.\n\nThe paper also provides numerous examples of images and the corresponding glimpses that were identified by the policy. A visual inspection of these examples provides convincing evidence that the policy is focusing on what we, as human, would consider the most important parts of each image. However, I would have preferred a less anecdotal approach. Perhaps the authors could, for example, evaluate how masking the glimpses identified by the policy impacts the accuracy of the prediction, and compares this to the impact of masking random parts of the image. \n\nLast but not least, the paper attempts to quantify the impact of their per-feature regularization mechanism. Their analysis demonstrates that this technique is a key element to the improved performance compared to previous hard attention work. \n\nThe paper is very well written. It is well organized, which makes it a breeze to follow. It was a pleasure to read. Furthermore, the approach can either deliver increased accuracy given a fixed compute budget, or reduce the compute cost given a fixed accuracy target. This tradeoff can be controlled simply by dialing the number location attended to. As a result, and thanks to its impressive results, I believe the paper will get a lot of attention (pun intended)\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mentioned the main limitation of the work, which is that the number of glimpses to attend to needs to be tuned manually to achieve a desired level of accuracy, and needs to be retuned for each dataset to achieve optimal performance. They also mention a number of follow up directions to further strengthen the work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "WzNFw2bJf9j", "reviewer": "Reviewer_PX35", "summary": "The authors present a hard attention mechanism, called  TNet,  for hi-res image classification.  The methods employ a resolution cascade, where a feature network attends to only a few number of locations of each resolution. The non-differentiable attention sampling is optimized with a Reinfore-like learning rule. All feature vectors are simply pooled for final classification, thus avoiding overhead from recurrent processing.\nThe authors compare their algorithm with two other hard-attention methods, Saccader and DRAM, and one fully convolutional architecture, BagNet-77 on ILSVRC 2012 ImageNet. Results on the opening figure indicate  that TNet is superior to the other methods on the Pareto frontier of accuracy and GFLOPS per image.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality\nThe idea of doing hard attention is not new, and as such, the TNet architecture does not come with too many big surprises. Please note, that this is not a criticism. What makes the work original and distinctive is how the authors designed the architecture to benefit from multi-resolution hard (and thus sparse) attention, and, at the same time, maintain computational efficiency. That efficiency is achieved imho, by (partially, on every resolution) parallel processing of feature extraction, and by a not overly complicated attention sampling mechanism. Also the feature aggregation with simple averaging is certainly due to  computational efficiency rather than model expressiveness. \n\n## Quality\nThe approach is technically sound. The derivations of the learning rules follow the line of Reinforce. New and surprising to me was to replace $p(y_i|l^i,x_i,w)$ by an indicator function (line 140-141). Although this sounds plausible, I'd loved to get some motivation for it (or a reference).\n\nWhat's missing is an exploration of the edge cases of the hyper parameters $\\lambda_f$ and $\\lambda_r,\\lambda_c$ (except for $\\lambda_r=\\lambda_c=1$ in Sec. 5.5).  In your experiments $\\lambda_f=0.1$ is not big, so it could be, in principle, that the second term of the learning rules doesn't actually matter. Same for $\\lambda_f,\\lambda_r$, could be that $L_F^s$ (2a) is actually irrelevant.    \nShowing that there is a sweet spot $0<\\lambda_{\\\\{f,r,c\\\\}}<1$ would remove such concerns. \n\n## Clarity\nThe paper is well written and easy to follow. I found it extremely fast to get the key message, just from looking at Figs. 1 and 2.\nThe authors might consider moving some material from the supplementary to the main paper, in order to make the main paper self contained:\n* line 140: What is the baseline $b$. Please put supplementary (5) to the main paper.\n* Sec. 5.4, Fig. 4: In order to judge the results, it is necessary to know the *feature weighting module*, which is only explained in the appendix.\n*  What is the value of $M$? This linearly affects the GFLOPS/image, no? In the appendix (Sec. 1.2) I've found that actually no sampling is done. This should be elaborate in the main paper, as it may affect the correctness of the learning rule.  \n\nFor all results, please indicate whether the GFLOPS are with or without backward processing. Event better, provide both numbers.\n\n## Significance\nI think the work is potentially significant, as hires image processing is a real application. The authors could have added to the significance by addressing a really large scale problem (896x896 is not exactly gigantic). Also, the paper could be stronger by not just comparing with other hard-attention models, but also any other efficient image classification approaches. \nIf TNet would be still superior, then it would give a clear signal to practitioners to focus on this method. \n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors address limitations of their model in Sec. 6. I would add that a further limitation is that, by construction, the model can only attend hi-res regions that have been attended to in low-res too (I'm not sure if that's meant with *\"scale-space traversal to be bi-directional\"*). \n\nI'm okay with their perspective of the broader impact being in increased resource efficiency, given that in a broader study TNet still can push forward the Pareto frontier of Fig. 1b. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes reducing the computational cost of running inference on high resolution images by leveraging a multi-scale attention architecture. The method progressively employs higher resolution imagery to zoom in regions of interest that aid in discriminative training, while ignoring any computation on non-informative regions. The resulting model is tested on ImageNet, fMoW and CUB-200-2011 against several baseline architectures, including  Saccader, DRAM, BagNet, and EfficientNet and shows favorable performance in terms of computational cost versus accuracy. The reviewers did voice some concerns about the lack of testing on a high resolution imagery, which remain unresolved. Overall, the reviewers were favorable with this work, and the concerns were notable, and did not rise to a significant enough level. This paper will be accepted to the conference.

Decision: Accept (Poster)