Achieving Explainability in a Visual Hard Attention Model through Content Prediction
Authors: Samrudhdhi B. Rangrej
James J. Clark
Venue:  (2021)
Keywords: visual hard attention
glimpses
explainability
bayesian optimal experiment design
variational autoencoder

Abstract: A visual hard attention model actively selects and observes a sequence of subregions in an image to make a prediction. Unlike in the deep convolution network, in hard attention it is explainable which regions of the image contributed to the prediction. However, the attention policy used by the model to select these regions is not explainable. The majority of hard attention models determine the attention-worthy regions by first analyzing a complete image. However, it may be the case that the entire image is not available in the beginning but instead sensed gradually through a series of partial observations. In this paper, we design an efficient hard attention model for classifying partially observable scenes. The attention policy used by our model is explainable and non-parametric. The model estimates expected information gain (EIG) obtained from attending various regions by predicting their content ahead of time. It compares EIG using Bayesian Optimal Experiment Design and attends to the region with maximum EIG. We train our model with a differentiable objective, optimized using gradient descent, and test it on several datasets. The performance of our model is comparable to or better than the baseline models.

Under review as a conference paper at ICLR 2021
ACHIEVING EXPLAINABILITY IN A VISUAL HARD ATTENTION MODEL THROUGH CONTENT PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
A visual hard attention model actively selects and observes sequence of subre- gions in an image to make a prediction. Unlike in the deep convolution network, in hard attention prediction. However, the attention policy used by the model to select these re- gions is not explainable. The majority of hard attention models determine the attention-worthy regions by first analyzing a complete image. However, it may be the case that the entire image is not available the beginning but instead sensed gradually through series of partial observations. In this paper, we design an efficient hard attention model for classifying partially observable scenes. The at- tention policy used by our model is explainable and non- non-parmmttric. The model estimates expected information gain (EIG) obtained from attending various re- gions by predicting their content ahead of time. It compares EIG using Bayesian Optimal Experiment Design and attends to the region with maximum EIG. We train our model with a differentiable objective, optimized using gradient descent, and test it on several datasets. The performance of our model is comparable to or better than the baseline models.
explainable which regions of the image contributed the
INTRODUCTION
Though deep convolution networks achieve state of the art performance on the image classification task, it is difficult to explain which input regions affected the output. A technique called visual hard attention provides this explanation by design. The hard attention model sequentially attends small but informative subregions of the input called glimpses to make predictions. While the attention mechanism explains the task-specific decisions, the attention policies learned by the model remain unexplainable. For example, one cannot explain the attention policy of caption generation model that correctly predicts the word 'frisbee' while looking at a region far from an actual frisbee (Xu) al. (2055)).
The majority of hard attention models first analyze a complete image to locate the task-relevant subregions and then attend to these locations make predictions (Ba et al. (2014); Elsayed et al. (2019). However, in practice, we often do not have access to the entire scene, and we gradually attend the important subregions to collect task-specific information. At each step in the process, we decide the next iitention-worthy location based on the partial observations collected so far. The explainable attention policies are more desirable under such partial observability.
Pioneering work by Mnih et al. (2014) presents a model that functions under partial observability but their attention policies are not explainable. They train their model with the REINFORCE algorithm (Williams) 1992), ), which is challenging optimize. Moreover, the model's performance is affected adversely if the parameterization classification model with unimodal Gaussian policy Iearns to attend the background region in the middle the two objects (Sermanet et al. (2014)).
the attention policy is not optimal. For example, an object
This paper develops a hard-attention model with explainable attention policy for classifying im- ages through a series of partial observations. We formulate the problem of hard attention as a Bayesian Optimal Experiment Design (BOED). A recurrent model finds optimal location that gains maximum expected information about the class label and attends to this location. To estimate expected information gain (EIG) under partial observability, the model predicts content of the un-

---

Under review as a conference paper at ICLR 2021
seen regions based on the regions observed so far. Using the knowledge gained by attending various locations in an image, the model predicts the class label.
To the best of knowledge, ours is the first hard attention model that is entirely explainable un- der partial observability. Our main contributions are as follows. First, our attention policies are explainable by design. One can explain that the model attends specifie location because it expects the corresponding glimpse to maximize the expected information gain. Second, the model does not rely on the complete image to predict the attention locations and provides good performance under partial observability. Third, the training objective is differentiable and can be optimized using stan- dard gradient backpropagation. We train the model using discriminative and generative objectives to predict the label and the image content, respectively. Fourth, our attention policy is non parameeriic and can be implicitly multi-modal.
2 RELATED WORKS
A hard attention model prioritizes task-relevant regions to extract meaningful features from an input. Early attempls to model attention employed image saliency as priority map. High priority regions were selected using methods such as winner-take. (Koch & Ullman (1987); [Itii et al. (1998); Itti & Kocc| (2000)), searching by throwing out all features but the with minimal activity (Ahmad 1992 ) and dynamic routing of information (Olshausen et al. (1993)).
Few used graphical models to model visual attention. Rimey & Brown (991) used augmented hidden Markov models to model attention policy. [arochelle & Hinton (010) used a Restricted Boltzmann Machine (RBM) with third-order connections between attention location, glimpse, and
the representation of scene. Motivated by this, ZZhnng et al. (2015) proposed an autoregressive model compute exact gradients, unlike in an RBM. Tang et al. (2014) used an RBM as a generative model and searched for informative locations using the Hamiltonian Monte Carlo algorithm.
Many used reinforcement learning to train attention models. Paletta al. (2005) used Q-learning with the reward that measures the objectness of the attended region. Denil et al. (2012) estimated rewards using particle filters and employed policy based on the Gaussian Process and the upper confidence bound. Butko & Movellan (2008) modeled attention as a partially observable Markov decision process and used a policy gradient algorithm for learning. Later, Butko & Movellan (2009) extended this approach to multiple objects.
Recently, the machine learning community use the REINFORCE policy gradient algorithm to train hard attention models (Mnih et al. (2014); Ba et al. (2044); ;; Xu et al. (015); Elsayed et al.) 20199). Among these, only Elsayed et al. (2019) claims explainability by design. Other works use EM-
style learning procedure (Ranzato (0144). wake-sleep algorithm (Ba et al. (2015)). voting based
region selection (Alexe et (2012), and differentiable models (Giregor et al. (0155)
) Jaderberg
et al. 2015); Eslami et al. (2016).
Among the recent models, Ba et al. (2014): Ranzato (2014; Ba et al. (2015) look at the low- resolution gist of input the beginning. and Xu et al. (2015); Elsayed et al. (2019); Gregor ct al. (2015): Jaderberg et al. (2015) Eslami et al. (2016) consume the whole image to predict the locations to attend. In contrast, our model does not look at the entire image low resolution or oth- erwise. Moreover, our attention policies are explainable. We can apply our model in wide range of scenarios where explainable predictions are desirable for the partially observable images.
MODEL
In this paper, we consider a recurrent attention model that sequentially captures glimpses from an image æ and predicts a label y. The model runs for time t 0 to T' 1. It uses a recurrent net to maintain a hidden state h that summarizes glimpses observed until time f 1. At time t, predicts coordinates I based on the hidden state h,- and captures square glimpse 9i centered at 4 in an image x, i.c. 9t = g(x, 4). It uses 9r and 4 to update the hidden state to h, and predicts the label y based the updated state hT-
2

---

Under review as a conference paper at ICLR 2021
91.4
R
p(nl)
ft
a(
9
s
(
(b)
Figure 1: A recurrent attention model sequentially observes glimpses from an image and predicts a class label. At time i, the model actively observes a glimpse 9: and its coordinates 4. Given Ir and 4, the feed-forward module F extracts features fa, and the recurrent module R updates a hidden state to he Using an updated hidden state 4,, the linear classifier C' predicts the class distribution p(y ht). At time 1+1 the model assesses various candidate locations before attending an optimal one. It predicts p(ylg,4, I, ft)) ahead of time and selects the candidate / that maximizes KL(p(ylg g. by he)llp(()))))
(a) The model predicts the content of g using a Partial VAE to compute p(ylg, 1, h2) without attending to the glimpse g. The normalizing flow-based encoder S predicts the approximate posterior q=2)11)), and the decoder D) predicts an image from a sample ~ ((=(/y). The model uses glimpses from the predicted image z to evaluate p(ylg. h..). Dashed arrows show a path to compute the lookahead class distribution p(ylg. L, ft). (b) Alternatively, the model predicts f. a feature representation of the image x. In f. the features at location I are the features of the glimpse at location The model uses f()) to predict the lookahead class distribution p(y (g, 1, h2) Ro (())f()),)) Predicting glimpse content in feature space shortens the lookahead path.
3.1 ARCHITECTURE
As shown in Figure [[), our model comprises the following three building blocks. recurrent feature aggregator (/ and I?) maintains hidden state h,, A classifier (C) predicts the class prob- abilities p(y(he). A normalizing flow-based variational autoencoder (S and D) predicts a complete image given the hidden state hy; flow-based encoder S predicts the posterior of a latent variable from he and a decoder D predicts a complete image from z. The BOED, as discussed in section 3.2 uses the predicted image find an optimal location attend the next time-step. To distinguish the predicted image from the actual image, let us call the former E. Henceforth, we crown any quantity derived from the predicted image á with (") ("). Next, we provide details about the three building blocks of the model, followed by discussion of the BOED in the context of hard attention.
3.1.1 A RECURRENT FEATURE AGGREGATOR
Given a glimpse gt. and its location 4, a feed-forward module extracts features /: = F(g., 4t). and a recurrent network updates hidden state h R(h fc). Following Mnih et al. (2014). we define F(ge,4, 4)) = BN ReakyReLU (F,()) + Fi(4))) where F, and Fi are deep networks, and R(1-11 ft) = (Leakyyyy ReLU Linear(h+-i) + Linear(f:)). Here, BN is a BatchNorm layer [Toffe & Szegedy 2015)) and LN is a LayerNorm layer (Ba et al. (2016).
3.1.2 A CLASSIFIER
At each time-step I, a linear classifier predicts the distribution p(y(hx) = C(h,) from a hidden state hr. As the goal of the model is to predict a label y for an image x, learn distribution p(ylh) by minimizing the minimization of the following cross- coos-eyy loss.
[[((())))]]p(((((t))) Optimization of this KL divergence is equivalent to
CcE(t) = =((y(=) log(p(y(k.))
(I)
3.1.3 A PARTIAL VARIATIONAL AUTOENCODER
We adapt a variational autoencoder (VAE) to predict the complete image x from the hidden state he. A VAE learns joint distribution between the image x and the latent variable given hr 2(14) = p(x|2/ <)p(2 An amortized encoder infers the posterior q(2|22, h4), which is an approximation of the true posterior p(z]ær, ht), and decoder infers the likelihood p(æ]2z) The training of VAE requires optimizing the Evidence Lower Bound (ELBO), which involves calculating [T h4) (=)b)))))) (Kingma& & Welling (003)). As the hard attention model does not observe
3

---

Under review as a conference paper at ICLR 2021
the complete image it cannot estimate q(2 2,, he). Hence, we cannot incorporate the standard VAE directly into a hard attention framework.
At the time t, we separate image into two parts, Or the set of regions observed up to t, and 24 - the set of regions as yet unobserved. Ma et al. (2018) observed that in a VAE, o, and u, are conditionally independent given 2, i.e. p(=|=) = p u4 ())))) They predict u, independently from the sample & ~ g(2)0r), while learning the approximate posterior ((=|0=) by optimizing the ELBO on log(p(o)))- They refer the resultant VAE as Partial VAE. Recall that the hidden state he calculated by our attention model is a summary of the glimpses observed up to 4, which is equivalent or, the of observed regions. Hence, we can write ((2(0r) as (2((he) in the ELBO of the Partial VAE.
CPV = Bo(alo.) log(p(o:(=)) l[((
= Bg(=lh.)) log(p(o,1=)) - K [L(g(2)h))lp((z)
(2)
In a Partial VAF, p(a, 2//4) p(w/l=))(po)) (2)22)) We implement a decoder D that predicts the complete image given the sample 2 (2))1)) Let my be binary mask with value 1 for the pixels observed by the model up to & and 0 otherwise; hence, O, = m, O where @ is an element-wise multiplication. We write the log- log-likelihood in equation 2 Jusing the mask me as follows.
log(p(o:1=)) 0.5 Σ mm D(z) m, C x -0. Σ m. O (D(z) x/2
(3)
the prior (() is Gaussian distribution with zero mean and unit variance. To obtain expressive posterior q=)//g), we use normalizing flonsmKingma et al. (2016). As an explicit inver- sion of the flows is not required, we use ruto-regressive Neural Spline Flows (NSF) (Durkan et al.) (2019) and efficiently implement them using single feed-forward network with masked weights as in De Cao et al. (2019). Between the two flow layers, we flip the input (Dinh et al. (2016) and normalize it using ActNorm (Kingma & Dhariwal (2018). In Figure aa). the flow-based encoder S infers the posterior q(2l/t)) S(;)). As mentioned earlier, we refer to the prediction from the Partial VAE as ã. The BOED uses I to find an optimal location to attend.
In equation 2
3.2 BAYESIAN OPTIMAL. EXPERIMENT DESIGN (BOED)
The BOED evaluates the optimality of set of experiments by measuring the information gain in the interest parameter due to the experimental outcome. (Chaloner & Verdinelli (1995)) In the context of hard attention, an experiment is to attend a location and observe a corresponding glimpse g g(x, d). An experiment of attending a location / is optimal if it gains maximum information about the class label y. We can evaluate the optimality of attending a specific location by measuring several metrics such as feature variance (Huang et (20T8)) uncertainty in the prediction (Melville ct al. (2004)), ) expected Shamnon information (Lindley ( (9566). For a sequential model, information gain K KL ylg, h4-1) h (1pphhhh))))) is ideal metric. It measures the change in the entropy of the class distribution from one time-step to the next due to observation of a glimpse g at location / (Bernardo (1979): Ma et al. (2018))
The model has to find an optimal location to attend at time before observing the corresponding glimpse. Hence, we consider an expectation of information gain over the generating distribution of An expected information gain (EIG) is also a measure of Bayesian surprise (Itti & Baldi (2006);
Schwartenbeck et al. )2013).
EIG(I) E. peil PP (T.plll (, h-
(4)
Inspired by Harvey et al. (2019), we define the distribution p(g]t, h1- as follows.
p(g1t, h4-1) Ey(s h, 1)()p,,
(5)
Here, 6(-) is delta distribution. As discussed in the section 3.1.3 the flow-based encoder predicts the posterior q()(/t,-)) and the decoder D predicts the complete image ã. g(-) extracts glimpse located at 1 in the predicted image ă = D(z). Combining equation 4 and equation 5 yields.
(1-04,,11111pp/( EIG(I) Eg KL (o--
(6)
We also tried an alternate method predict the image x given h2 as discussed in section
4

---

Under review as a conference paper at ICLR 2021
To find an optimal location to attend at time t. the model compares various candidates for 4. It pre- dicts EI)( for each candidate i and selects an optimal candidate as 4, i.. 4 = arg max; (IG(l).
When the model is considering a candidate I, it predicts f = ((g(2,)) 1), 1), he = R( and p( = C(ha). It uses the distribution p(y(ht) = p(vlg(D(2).,), h.-1) to predict EIG in equation We refer to p(y().) as the lookahead class distribution computed by anticipating the content at the location ahead of time. In Figure (Ta) a). the dashed arrows show lookahead step.
3.2.1 EFFICIENT COMPUTATION OF EIG
Convolutional implementation: To compute EIG for all locations simultaneously, we implement all modules of our model with convolution layers. The model computes EIG for all locations as a single activation map single forward pass. An optimal location is equal to the coordinates of the pixel with maximum value in the EIG map.
During the lookahead step, the feature extractor F computes a feature map f from the predicted image I. However, we can use the decoder to directly predict the feature map f. Predicting f instead of ō achieves two goals. First, the time to compute ETG is reduced as the model does not require a feature extractor F during the lookahead step (see Figure [b)). Second, the Partial VAE does not have to produce unnecessary details that may later be thrown away by the feature extractor F, such the exact color of pixel.
Feature generation:
In most cases, the number of elements in the feature map f is greater than the elements in ã. The model requires more memory resources and parameters to predict f. So instead, our model predicts a subsampled feature map that contains features for every nh glimpse. Consequentially, the model computes EIG map at low resolution and finds optimal glimpses from a subset of glimpses separated with stride equal to n.
When the decoder predicts a feature map f instead an image ž, the feature likelihood p(fin (z) replaces the likelihood p(o, lz) in equation 2 fix are the features of the glimpses observed by the model until time t. Similar to equation 3] the corresponding lig-liiel is computed as follows.
log(p(fi:
0.5
Σ me + [D(()) fl?
Here, f is a map containing the features of all glimpses. As f a feature map of strided glimpses, subsample the mask m, and the feature map / in the above lielliell.ood.
4 TRAINING AND TESTING
The training objective is L (PAA(() + ACCE((), where À is hyperparameter. The training procedure is given in section B We use only one sample & ~ q(2(h;) to estimate the EIG map during the training. We found this to work well in practice. We do not compute gradients and do not update the statistics of normalization layers during the lookahead step. When the decoder predicts the feature map instead of the image, we pre-train modules F, R, and C for few epochs with random glimpses. This provides a reasonable target f for the log-likelihood in equation 7 We naturally achieve exploration early in the training when the model produces noisy output and computes noisy EIG maps. As the training progress and the predictions become accurate, the exploitation begins. The testing procedure is shown Algorithm I Unlike in the training phase, we use P samples of 2 to estimate the EIG accurately during the prediction phase.
CPVAn(!)
Algorithm Test procedure for the model shown in Figure I
1: Randomly sample lo: Capture go at lo. compute fo. ho and p(ylho)
2: for / E (1,..., - do
D. ' the time budget D is the sample budget b equation
3: Sample 2; ~ q(2 (2)) and predict f); E P-13
4: Compute it. p(y (T)) and EIG = 1o E,
5:
4 arg max FTG)
6:
Capture 9t at 4; Compute ft. h, and p(y)(z)
7: end for
5

---

Under review as a conference paper at ICLR 2021
1.0
0.
Accumcy
OIN
04
Soft Attendon EXPA
0.6
JAM
GRAM
Random
03
0.4
Time
Time
(a)
(b)
0.8
0.6
0.5
0.6
0.5
0.3
0.4
62
0.3
0.0
Time
Thme
(c)
(d)
(c)
Figure 2:
The classification accuracy as a function of time. A CNN and a soft attention model observe the entire image predict the class label. EXPAC, RAMi et al. (2014)), and GRAM sequentially attends glimpses in the image and predict the class label. EXPAC and RAM never observe a complete image and attend the first glimpse at random location. GRAM observes gist of an input at the beginning and predicts glimpse locations from / = O. (a) MNIST (b) SVHN (c) CIFAR- (d) CIFAR-100 (c) CINIC-10.
EXPERIMENTS
We refer to our model as 'EXPAC' which stands for EXPlainable Attention with Content prediction. We evaluate EXPAC MNIST (LeCun et al.] (1998)), SVHN (Netzer et al. (21TT). CIFAR-1O, CIFAR- 100 (Krizhessky et al. (2009)), and CINIC-I0 (Dallww et al. (2018)) datasets. The MNIST and the SVHN datasets consist of gray-scale and color images of the 0-9 digits. respectively. The CINIC-10, CIFAR-10 10 and CIFAR- 1-00 datasets are composed of color images of real-world objects the natural setting, categorized into 10, 10 and 100 classes, respectively.
For the MNIST dataset, we use the model shown in Figure [a). This model predicts image ă and EIG full resolution and finds optimal glimpse from a set of all overlapping glimpses. We use the model shown in Figure (b) for the remaining datasets. This model predicts f and ETG for set of glimpses separated with stride equal to n. We do not allow this model to revisit glimpses attended the past. All models run for T = time-steps and sense glimpses of size 8 x 8 with stride 4. The hyperparameter À in the training objective is 500, and the sample budget P is 20 for all experiments. We did not observe any performance gain with P > 20. Refer to section and for the remaining details.
5.1 EXPAC PREDICTS CLASS-LABELS ACCURATELY
We compare EXPAC with five baseline models Figure The RAM is state-of-the-as model for hard attention that classify partially observed images (Miih et al. 2014)). Our implementation of RAM has similar structure for the feature extractor F. the recurrent net R, and the classifier as EXPAC. Instead of the Partial VAE, RAM has controller that learns a Gaussian attention policy. We extend RAM by feeding it with a low-resolution gist of an input at the beginning. To ensure that gist is used only to predict glimpse locations and not the class labels, the controller receives input from a separate recurrent net that runs parallel to the recurrent net R. We predict the initial hidden state of this additional recurrent net from the gist. We refer this model Gist-RAM GRAM. We also consider a baseline model that attends glimpses on random locations and predicts class labels. Note that EXPAC and the three baselines described far observe the image only partially

---

Under review as a conference paper at ICLR 2021
0.
0.8
0.4
DPPCI
E3NC
MMM
04
GRAM
0.2
3.
4
03
5
9
(a)
(b)
(c)
(d)
Figure 3: The classification accuracy function of time. iEXPAC interpolates EIG maps to find optimal glimpses that may overlap. (a) SVHN (b) CIFAR-10 (c) CIFAR-100 (d) CINIC-10.
through a series of glimpses. Additionally, we train a feed-forward CNN and a soft attention model that observe the entire image predict the class label. For the soft attention model, adapt the method proposed by JJtley et al. (201)).
The CNN outperforms all other models. The performance of hard attention models surpasses the soft attention model after the former observes sufficient number of glimpses. Unlike the hard attention models, the soft attention model exhausts its capacity tion weights for the uninformative regions. Among the hard attention models, EXPAC shows the best performance. All hard attention models perform better than the random baseline for the digit datasets. For natural image datasets, RAM and GRAM fall behind the random baseline. As observed by "Sermanet al. (0144). RAM does not scale well natural images, probably due the unimodal attention policy. Except for the SVHN dataset, GRAM's accuracy is less than RAM's accuracy, despite observing the input gist. We also observed that GRAM overfits the training data and does not generalize well to the test set.
computing features and atten-
We speculate that regularizing the feature space for classification very important for the hard atten- tion models. The gradients from the ELBO and the REINFORCE objective regularize this feature space in EXPAC and RAM, respectively. Morcover, the ELBO appears to be a better regularizer than the REINFORCE objective. Zheng et al. (2015) also observed that the use of generative loss regularizes the hard attention model. GRAM uses separate feature spaces for the classifier and the controller; hence the former is not regularized by the latter. GRAM overfits the train set and does not perform well on the test set due to a lack of regularization.
5.2 EXPAC PERFORMS WELL WITH THE INTERPOLATED EIG
To let EXPAC find and attend overlapping glimpses with unit stride, we assume smoothness prior and upsample the EIG map using bicubic interpolation to get EIG values for all overlapping glimpses. We refer to this model as interpolation- E.XPCC or iEXPAC. [eguuee]showe a comparison between EXPAC and iEXPAC. iEXPAC performs better than or comparable to EXPAC during initial time-steps. As time progresses, EXPAC observes more input regions than iEXPAC through strided glimpses and outperforms iEXPAC. However, iEXPAC still performs better than RAM and GRAM, which also attends the overlapping glimpses with unit stride. Next, we assess the policies used by these models.
5.3 EXPAC LEARNS USEFUL POLICY FOR CLASSIFICATION
We compared the accuracy of the hard attention models as function of the area covered in the image. We found that iEXPAC achieves an accuracy similar to the other models by covering less area in the image. The result suggests that iEXPAC observes few but informative regions (refer section D.I. However, this analysis provides only a narrow perspective on the policies used by different models, each model learns classifier that adapts to its policy. To better understand the policies, we cover the glimpses observed by the hard attention models and let the baseline CNN (from Figure 2 classify the occluded images. Accuracy should drop if the object of interest is occluded. This analysis is similar to the one performed by [lsaayed et al. (2019). The results for natural image datasets shown in Figure (refer section (D2 for the results on digit datasets). We do not observe any generalizable trend for GRAM. However, we observe that the accuracy drops faster by covering the glimpses attended by the RAM compared to iEXPAC. It appears that the

---

Under review as a conference paper at ICLR 2021
5:40
ESRO Sland
3.40
0:5
0.33
1:40
0.03
035
0.30
022
13
(a)
(b)
Figure 4: The classification accuracy of CNN as a function of the area occluded. CNN classifies images with the glimpses attended by various hard attention models occluded. (a) CIFAR-10 (b) CIFAR- (c) CINIC-10.
6
8
0
6
6
2
6
06600000
8
6
4
1=2
6
/
7
/
7
8
33333533
3
8
8
4
8
9
9
9
9
13
/
/
/
/
/
7
1
3
13
7
7
7
7
9
9
9
2
7
7
7
7
7
7
2
7
7
9
9
9
7 9 9
9 9
2
2
2
2
2
2
33
33
7
9
8
7
8 9
1-6
2
2
2
2
2
2
2
2
2 M-6
3
3
3 (b)
3
1-6
7
7
9
8
9
7
9
(a)
(c)
Figure 5: Visualization of the predictions from the Partial VAE, In each panel, from left to the right: time step, glimpses observed until time eight samples of T. The content in & on the observed locations is consistent with the glimpses.
RAM attend to the object of interest more frequently than iEXPAC. However, iEXPAC achieves better performance. The use of EIG in BOED explains the behavior of iEXPAC. the attended glimpse provides evidence in favor of one class and the Partial VAE also generates features in favor of the same class, then EIG for areas characterizing that class is low. In this case, iEXPAC explores areas with high EIG, away from the regions approving of the believed class. Consequently, iEXPAC often attend regions outside the object of interest. Being a measure of Bayesian surprise, EIG promotes iEXPAC seek glimpses that seem novel according to the latest belief (Itti & Baldi (006); [Schwartenbeck et al. (2013)). During the process, iEXPAC may collect evidence favoring an actual class that may have initially appeared unlikely and change the decision.
5.4 VISUALIZATION OF THE EIG MAPS AND THE ATTENTION POLICY USED IEXPAC
We display the generated samples of I for MNIST dataset in Figure 5 For the remaining datasets, we predict f. Figure G shows the EIG maps and the optimal glimpses found by iEXPAC CIFAR- 10 images. First, notice that the EIG maps are multi-modal. Second, activity the EIG maps reduces the model gathers sufficient glimpses predict class label. In Figure a)), activity in EIG map is reduced as the model settles on a class 'Do''. Once discovers left car of cat at t = 3, the activity in the EIG map increases at t 4. The model changes its decision to 'Cat' once discovers the right ear at f = 5. In Figure (b), the model predicts class 'Airplane' at time t 3 once it discovers body of airplane. At the next time-step, the model expects reduced information gain on the areas where the remaining body of the airplane may lie. In Figure [(c), the model predicts class 'Deer' once discovers face of the animal at 4. Notice reduced activity in the EIG maps at t = 5 and 6.
5.5 EXPAC ATTAINS HIGHER ACCURACY WITH FLEXIBLE POSTERIOR: AN ABLATION STUDY
We investigate the necessity of a flexible distribution for the posterior g(2(/z) and, therefore, the necessity of normalizing flows in the encoder S. To this end, we model the posterior with unimodal Gaussian distribution and let S output mean and diagonal covariance of a Gaussian. We do not flow layers in this case. Figure 7 shows the result for natural image datasets. Refer Figure II for results on digit datasets. Modeling a complex posterior using normalizing flows improves accuracy during early time steps. The gap in the accuracy of the two reduces with time. The reason
8

---

Under review as a conference paper at ICLR 2021
H+
[-]
P-2
D
[-
1-5
t-6
1-0
P-3
Dog
Dog
Dog
Dog
Cwt
Airplane
Ship
ShiP
Airplane
Airplane
Airplane
Airllane
(b)
I+0
[-]
t+2
5
I+5
b-G
I+D
D-1
D=2
[ed
1-4
D-5
I-G
Airllane
(c)
(d)
Figure 6: Visualization of the EIG maps and the glimpses observed by iEXPAC CIFAR-IO 10 im- ages. The top row shows the entire image and the EIG maps for time t = 1 to 6. The bottom row shows glimpses attended by iEXPAC. The model observes the first glimpse We display the entire image for reference; iEXPAC never observes the whole image. (a-c) success cases (d) failure case.
random location.
0.80
0.75
0.75
0.5
0.70
0.05
Accuricy
0.80
0.60
0.55
0.55
0.50
(CurS)
0.45
Noama (Sues)
0.45
6
0.40
3
5
Time
Time
Time
(a)
(b)
(c)
Figure 7: The classification accuracy as a function of time. We compare performance of EXPAC
with and without using normalizing flows. (a) CIFAR-I0 (b) CIFAR-100 (c) CINIC-10.
being follows. Ideally, the Partial VAE should predict all possibilities of ă (or f) consistent with the observed region Oz- Initially, when EXPAC observes a small region, complex multi-modal posterior helps determine multiple plausible images from different classes. unimodal posterior fails to cover all possibilities. As time progress, EXPAC observes large region, and the associated possibilities decrease. Hence, the performance gap between complex and a uni-modal posterior decreases. Eventually, only single possibility of a complete image remains. Then, the performance of unimodal posterior meets the performance of complex posterior.
CONCLUSION
We presented a hard attention model that uses BOED to find the optimal locations to attend when the image is observed only partially. To find an optimal location without observing the corresponding glimpse, the model uses Partial VAE to predict the content of the glimpse. The model predicts the content either the image space or the feature representation space. The predicted content enables the model to evaluate and compare the expected information gain (EIG) of various candidate locations, from which the model selects a candidate with optimal EIG. The model predicts feature representations and EIG of a subset of glimpses for computational efficiency. We interpolate EIG maps to let the model find optimal glimpses that may overlap. The attention policy used by our model is non parametric, multimodal, and explainable. We trained our model with differentiable objective and tested on five datasets. We found that our model attends glimpses highly relevant to the classification task and achieves better performance than the other baseline models.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Subutai Ahmad. Visit: a neural model of covert visual attention. In Advances in neural information processing systems, pp. 420-427, 1992.
Bogdan Alexe, Nicolas Heess, Yee W Teh, and Vittorio Ferrari. Searching for objects driven by context. In Advances in Neural Information Processing Systems, pp. 881-889, 2012.
Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. Multiple object recognition with visual attention. arXiv preprint urXiv:1412.7777 1412.775, 2014.
Jimmy Ba, Russ R Salakhutdinov, Roger B Grosse, and Brendan J Frey. Learning wake-sleep recurrent attention models. In Advances in Neural Information Processing Systems, pp. 2593- 2601, 2015.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.066 2016.
José M Bernardo. Expected information as expected utility. the Annals of Statistics, pp. 686-690, 1979.
Nicholas J Butko and Javier R Movellan. I-pomdp: An infomax model of eye movement. In 2008
7th IEEE International Conference on Development and Learning, pp. 139-144. IEEE, 2008.
Nicholas Butko and Javier R Movellan. Optimal scanning for faster object detection. 2009
IEEE Conference Computer Vision and Pattern Recognition, pp. 2751-2758. IEEE, 2009.
Kathryn Chaloner and Isabella Verdinelli. Bayesian experimental design: review. Statistical
Science, pp. 273-304, 1995.
Luke N Darlow, Elliot Crowley, Antreas Antoniou, and Amos Storkey, Cinic- 10 is not imagenet or cifar-10. arXiv preprint arXiv: 1810.03505 2018.
Nicola De Cao, Ivan Titov, and Wilker Aziz. Block neural autoregressive flow. arXiv preprint 1rXi::1904.04676, 2019.
Misha Denil, Loris Bazzani, Hugo Larochelle, and Nando de Freitas. Learning where to attend with deep architectures for image tracking. Neural computation, 24(8):2151-2188 2012.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: 08803, 2016.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems, pp. 7511-7522, 2019.
Gamaleldin Elsayed, Simon Korbblith, and Quoe V Le. Saccader: improving accuracy of hard attention models for vision. Advances Neural Information Processing Systems, pp. 702- 714, 2019.
SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hin- ton, et al. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pp. 3225-3233, 2016.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, pp. 1704-1711 2018a.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:1807.01622. 2018b.
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. In ICML, 2015.
William Harvey, Michael Teng, and Frank Wood. Near-optimal glimpse sequences for improved hard attention neural network training. arXiv preprint urXiv: 05562, 2019.
10

---

Under review as a conference paper at ICLR 2021
Sheng-Jun Huang, Miao Xu, Ming-Kur Kun Xie, Masashi Sugiyama, Gang Niu, and Songcan Chen. Active feature acquisition with supervised matrix completion. In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 1571-1579, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 150.00360 2015.
Laurent Itti and Pierre F Baldi. Bayesian surprise attracts human attention. In Advances in neural information processing systems, pp. 547-554, 2006.
Laurent Itti and Christof Koch. A saliency-based search mechanism for overt and covert shifts of visual attention. Vision research, 40(10-11);1489- 1506, 2000.
Laurent Itti, Christof Koch, and Ernst Niebur. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on pattern analysis and machine intelligence, 20(1 12544 1259, 1998.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In Ad- vances in neural information processing systems, pp. 2017-2025, 2015.
Saumya Jetley, Nicholas A Lord, Namhoon Lee, and Philip HS Torr. Learn pay attention. In International Conference on Learning Representations, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69888 6980, 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114 2013.
arXiv preprint
Durk P Kingma and Prafulla Dhariwal. Glow:
Generative flow with invertible Ix1 convolutions. In Advances in neural information processing systems, pp. 10215-10224, 2018.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In Advances in neural information processing systems, pp. 4743-4751, 2016.
Christof Koch and Shimon Ullman. Shifts in selective visual attention: towards the underlying neural circuitry. In Matters of intelligence, pp. 115-141. Springer, 1987.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Hugo Larochelle and Geoffrey E Hinton. Learning combine foveal glimpses with third-order boltzmann machine. In Advances in neural information processing systems, pp. 1243-1251, 2010.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):228--2224 1998.
Dennis V Lindley. On a measure of the information provided by an experiment. The Annals of
Mathematical Statistics, pp. 986-1005, 1956.
Chao Ma, Sebastian Tschiatschek, Konstantina Palla, José Miguel Hernández-Lobato. Sebastian Nowozin, and Cheng Zhang. Eddi: Efficient dynamic discovery of high-value information with partial vae. arXiv preprint / /421 2018.
Prem Melville, Maytal Saah-Tsschansky, Foster Provost, and Raymond Mooney. Active feature- value acquisition for classifier induction. In Fourth IEEE International Conference Data Min- ing (ICDM'04) pp. 483-486. IEEE, 2004.
Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In Advances in neural information processing systems, pp. 2204-22222 2212, 2014.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011. 1.

---

Under review as a conference paper at ICLR 2021
Bruno A Olshausen, Charles H Anderson, and David C Van Essen. A neurobiological model of vi- sual attention and invariant pattern recognition based on dynamic routing of information. Journal 1993.
of Neuroscience, 13(11):4700-
Lucas Paletta, Gerald Fritz, and Christin Seifert. Q-learning of sequential attention for visual object recognition from informative local descriptors. In Proceedings of the 22nd international confer- ence on Machine learning, pp. 649-656, 2005.
Marc' Aurelio Ranzato. On learning where look. arXiv preprint 14055:4455.5433 2014.
Raymond D Rimey and Christopher Brown. Controlling eye movements with hidden markov models. International Journal of Computer Vision, 7(1):47-65, 1991.
Philipp Schwartenbeck, Thomas FitzGerald, Ray Dolan, and Karl Friston. Exploration, novelty, surprise, and free energy minimization. Frontiers in psychology, 4:710, 2013.
Pierre Sermanet, Andrea Frome, and Esteban Real. Attention for fine-grained categorization. arXiv preprint arXiv:1412.7000 2014.
Charlie Tang, Nitish Srivastava, and Russ Salakhutdinov. Learning generative models with visual attention. In Advances in Neural Information Processing Systems, pp. 1806-1816, 2014.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992.
Kelvin Xu, Jimmy Ba. Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference machine learning, pp. 2057, 2015.
Yin Zheng, Richard S Zemel, Yu-Jin Zhang, and Hugo Larochelle. A neural autoregressive approach to attention- recognition. International Journal of Computer Vision, 113(1):67-7,, 2015.
12

---

Under review as a conference paper at ICLR 2021
A NEURAL ARCHITECTURE
We implement EXPAC for MNIST dataset based on Mnih et al. (214)). We replace ReLU with LeakyRe and add BN in their architecture. Table shows the neural architecture of EXPAC for color datasets. We implement linear layers using 1 x convolution layers. Note that all modules use an indispensable number of layers. Fi is the shallowest network to learn the non-linear repre- sentation of the input. Fg uses the least possible layers to achieve the effective receptive field equal to the area of a single glimpse. The decoder D uses the smallest number of Conerrannspooee and Conw2d to generate the feature maps of required spatial dimensions and refine them based on the global context. We use linear classifiers for SVHN, CIFAR-I0 and CINIC-I0 datasets and use two-layer classifier for CIFAR-100. The dimensionality of f. h and for various datasets are mentioned in the 2
Module
Architecture
Fi
Conv2d(k: 1) LeakyReLU - BN - Conu2d(k: = 1) (Comv2d(k 3) Lea - BN) Comv2dl( 2)
Recurrent feature
Fg
3 x
aggregator
F(g.44) 4)
(F + Fi(l)))
R(hi-1.f1)
LN(LeakyRe (Cono22(k = 1)( + Comv2d(k =))ffi))
Classifier
C
Conv2d(k 1)
S
x {ActNorm permute - NSF)
Partial
VAE
D
3 [ConuTransposeddif 3) LeakyReLU - BN
4 x {Cona2d(k = p.p 1) ReakyRelU BN)
Table 1: Neural architecture of EXPAC for SVHN, CIFAR-10 and CINIC-10 datasets. k kernel_size, p = padding.
f
h;
2
MNIST
256
256
256
SVHN
128
512
256
CIFAR-10
128
512
256
CIFAR-100
256
1024
256
10
128
512
256
Table 2: The size of f. hy and 2.
B OPTIMIZATION DETAILS
We train the model using Adam optimizer (Kingma & Ba (2014) with a learning rate 0.001 and default 31 and 3g. and divide the learning rate by models on single Tesla V100 GPU with 16GB of memory for approximately a day. The training procedure for the model shown in Figure Tbb) is shown in Algorithm
plateau until convergence. We trained all
C ALTERNATE GENERATIVE MODEL
We try an alternate generative model for the image given ht At time f, we divide the image into two parts, or and uu, a set of regions observed by the model and a set of regions that are yet unobserved. The set of observed regions o; is equivalent to the glimpses observed by the model until time f, i.e 0₂ = {g1:, 41=}- Recall that the hidden state h4 summarizes these glimpses. Hence, we write p(z) (.,) = p(z(oz) p(u lo, )p(o) For p(u, (or) and p(or), we can learn two latent variable models which optimize the ELBO on log(p(u, (.)) and (pg.)( These ELBOs are similar to the ones used train Neural Processes (Garnelo et al. (00Th a)) and Partial MA al. 2018
13

---

Under review as a conference paper at ICLR 2021
Algorithm 2 Training procedure for the model shown in Figure []b
while not converged do
2:
C=
3:
for E (O..... 1) do
the time budget
4:
if t then
5:
Randomly sample lo else with gradient:
6:
7:
Compute hes phy and EIG
Use f and h4-1 computed at b equation
8:
4 = arg =aa Ea
9:
end if
10:
Capture 9t at 4,; Compute ft, h4 ply(h)), ((2)he) and 11: L= CPVAR + ALCr
equation 2
12:
end for
13:
Compute gradients
#c and update model parameters 0 20
14:
end while
respectively.
NNP oo log(p(u/2,,0)) - (llg(al.,,))lllttt)))
(1) (og(p(u//,,)))) h.)) - (o)
(8)
LAAA (o(x(o)) log(p(o:) =)) {L]g(=oo))ll(e))
(e()))))) log(p(o,)z)) [Llg(]]
C.I IMPLEMENTATION
We implement single VAE that takes image and hidden state he as input and reconstructs the image x. The design of this VAE is an extension of Partial VAE from Figure T with a conditional
encoder E and normalizing flows S. We train this VAE with combination of ELBOs in equation
and equation 9 The prior p(z) is a Gaussian distribution with zero mean and unit variance. A conditional encoder E infers the posterior ( he). We conditional normalizing flows to establish a relation between (()h)) and ((2(/4), i.e. 2 = 5(2,ha).
a( h.) q( (==(h+) det 05 (5((,h))
(10)
az
As in Figure I the flowd encoder S and the decoder D predicts the approximate posterior (22))) and the complete image H, respectively. Given the binary mask m, discussed in section we compute the likelihoods in equation 8 and equation aas follows.
1 Σ
log(p(or12) = 2 my O (D(x) - xx12
(11)
log(p(up|z,hh)) Σ m.) (D(S(2,h.)) - x/2
(12)
C.2 PERFORMANCE ANALYSIS OF THE ALTERNATE GENERATIVE MODEL
Here compare model's accuracy when the generative model is Partial VAE versus combi- nation of Partial VAE and Neural Processes. We also train the alternate VAE with just the ELBO of Neural Processes. Figure X shows the result. We do not find single generative model that performs well across datasets. We selected Partial VAE for our model as it does not use an encoder E and S, which require additional parameters.
D ADDITIONAL RESULTS
D.1 EXPAC ATTENDS TO GLIMPSES RELEVANT TO CLASSIFICATION
We assess the usefulness of the glimpses attended by EXPACЛEXPAC, RAM, and GRAM for the classification task. Figure 9 shows the models' accuracy function of the observed in
14

---

Under review as a conference paper at ICLR 2021
1.0
0.0
0.8
Accumtt
EXACEXPAO
PVAENP
0.4
Tima
(a)
(b)
(c)
Figure 8: Classification accuracy as a function of time. We test various generative models predict the image content. (a) MNIST (b) SVHN (c) CIFAR-10.
1.0
1.0
0.0
0.7
0.5
EXPAC/Ours)
EXPAA
RAM
RAM
GRAM
GHAM
Area
(a)
(b)
0.6
0.7
0.5
0.6
0.4
0.5
0.3
0.4
0.1
0.2
0.1
0.2
0.3
0.4
0.1
0.2
033
0.4
Area Covered
Area Covered
Area Covered
(d)
(e)
Figure 9: Classification accuracy of the hard attention models a fanction of the area covered. (a)
MNIST (b) SVHN (c) CIFAR-10 (d) CIFAR-100 100 (e) CINIC-10.
image. Compared to the RAM and GRAM, EXPAC/EXXAC achieves high accuracy by observing
less area. The results indicate that EXPACЛEXPAC observes regions that are more relevant to classification task than the other two models. Attending few but task-relevant regions lead to better performance.
D.2 EXPAC LEARNS A USEFUL POLICY FOR CLASSIFICATION
To get an insight into the attention policies used by different models, occlude the glimpses attended by the hard attention models and let a baseline CNN predict the class label from an occluded image. We expect drop in accuracy when the foreground object is occluded. The results for natural image datasets are discussed in section 5.3 Here, we discuss results for the digit datasets. Unlike in the case of natural images, CNN's accuracy for EXPACЛEXPAC drops at greater or similar rate RAM and GRAM. In natural images, iEXPAC explores background regions with high EIG to find evidence for different classes. In digit datasets, the foreground object is at the center and not related to the surrounding. Hence, EXPAC learns to predict low EIG on the background and always finds an optimal glimpse on the foreground. The accuracy of CNN drops as EXPAC attends informative glimpses on the foreground. Notice that the accuracy of CNN drops faster for GRAM on the SVHN dataset. The trend suggests that GRAM finds the most informative glimpses. The GRAM also achieves better performance than the RAM for this dataset, shown in Figure] 2
15

---

Under review as a conference paper at ICLR 2021
EXPACIOUrS]
DEXPACLDurS!
0.8
RAN
RAM
Accuracy
Accuracy
0.1
0.5
Occluded
Occluded
(a)
(b)
Figure 10: The classification accuracy of a CNN as function of the area occluded. A CNN clas- sifies images with the glimpses attended by various hard attention models occluded. (a) MNIST (b)
SVHN.
0.9
Accuracy
0.6
EXPAC flows [Ours) EXPAC no flows (Ours)
Time
(a)
(b)
Figure 11: The classification accuracy as a function of time. We compare performance of EXPAC with and without using normalizing flows. (a) MNIST (b) SVHN.
However, iEXPAC outperforms GRAM. Unlike GRAM, iEXPAC learns better feature space for classification due to the regularization provided by the generative objective during the training.
16


Reviews:
[{"review_id": "tUC2tnTcoj", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n**Summary:**\n \nThis paper follows a less explored strategy for achieving explainability via hard attention. They proposed a recurrent architecture which sequentially observe regions (glimpse) from an image. To decide where to look next, the model maintains a hidden state and use it to estimate the full image (or features of the image). This \"content prediction\" module allows the model to look ahead and make a decision based on the expected information gain (EIG) over different locations. The objective function (i.e. partial VAE loss and classification loss) in this system is differentiable thus the system can be trained with gradient descent. The authors validated the system on several benchmarks and show comparable performance with baselines.\n\n**Reasons for score:**\n\nThe system seems extremely complicated to me, as it involves multiple components and each component by themselves is very complex. However the output of the system is not so appealing either in performance or explainability. Probably I missed something but I don't quite understand the advantages of the proposed system.\n\n**Pros:**\n\n1. The core idea for training the attention policy is intuitive, as the information gain is a natural choice for determining the location of next glimpse. It's also appealing that the predicted content can provide sufficient signals to estimate the information gain.\n2. The equation is clear and makes the paper easy to follow.\n3. The careful analysis of the experiments is very informative.\n\n**Cons:**\n\n1. In general, my biggest concern about this paper is the complexity of the EXPAC system with the moderate performance. It seems that all benchmark datasets used in the paper are not so difficult (e.g., 10-way classification with 32x32 images) and the performance of the proposed system is still far from satisfactory. Therefore, it's questionable whether the system could be scaled to even more challenging (but more practical) datasets like ImageNet. Also I'm curious about the robustness of training such an intricate system. \n2. Regarding explainability, there are quite a few methods (not limited to hard attention) that target the same goal, such as Grad-CAM. However this paper seems to only compare with RAM (and a gist-initialized variation of GRAM). Even if the core algorithm might be different, it's still good to compare with other designs. \n(Selvaraju, Ramprasaath R., et al. \"Grad-cam: Visual explanations from deep networks via gradient-based localization.\" Proceedings of the IEEE international conference on computer vision. 2017.)\n3. It seems to me that the content prediction module ($S$ and $D$) is critical as the predicted $\\tilde{x}$ is used for \"lookahead\" to estimate the next $l$. However the predicted $\\tilde{x}$ is never shown in the paper. I think it would be more straightforward to  better understand the performance of the system by comparing the predicted content with the original.\n\n\n**Questions during rebuttal period:**\n\nPlease address my questions in the cons section.", "extra_scores": {}, "extra_text": {}}, {"review_id": "qovkXs5ewmG", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an alternative way to conduct hard attention. Specifically, it estimates the expected information gain (EIG) of attending various regions and at each timestep chooses the region with the maximum EIG. The proposed method is tested on image classification.\n\nIssues:\n1. what do you mean \"achieving explainability\" in the title? I don't quite get it. The visualization in Fig.5 shows only that the region selected in each timestep indeed has the maximum EIG. But how to interpret the explainability from the glimpse sequence is still confusing. I can hardly perceive the sequence using my knowledge. \n2. why the proposed method needs three building blocks is not well explained, especially the partial VAE. \n3. what is the data used to train the  partial VAE?\n4. no ablation study is conducted on these building blocks, such as the normalizing flow module in the partial VAE.\n5. I think image classification is not the best task to demonstrate the effectiveness of hard attention method. As shown in the experiments, simply using a standard CNN that takes the whole image as the input obtains the best result.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "p9ETLFbbs76", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a visual hard-attention image classification model. The difference to standard classification methods such as CNN is that the model provides an explainable inner structure by default, that can be inspected to see what the model focused on. The difference to other state-or-the-art hard-attention models is that this model is differentiable, allowing for more robust and stable optimization.\n\nOn a positive note, the presented method is sound and mathematically principled, and the description of it is complete and technically correct. The paper is also well written, well organized, and easy to read. The relevant related work is cited.\n\nHowever, the paper suffers from two major flaws. \nFirstly, the contribution of the proposed method with respect to other recent hard-attention models based on reinforcement learning it is not well motivated - other than that this model is differentiable. The last paragraph in the Related Work provide no statement whatsoever as to what the present method contributes over the latest methods in the literature. \nSecondly, the baseline hard-attention model in the experiments, (Mnih et al. 2014), is very old and it is not surprising that the proposed method outperforms it. A more interesting baseline would be a later hard-attention model such as (Elsayed et al. 2019). Moreover, the used datasets are all quite simplistic, and it would be more interesting with a more realistic one.\n\nDue to the above, the recommendation is Reject - but the authors are strongly encouraged to do experiments on more challenging data and compare to a newer baseline.", "extra_scores": {}, "extra_text": {}}, {"review_id": "E0xb-ecDxib", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThis paper proposed a new hard attention model for the image classification. They designed hard attention mechanism as a bayesian optimal experimental setting. Compare to other hard attention model, the policies of proposed hard attention can be explainable and differentiable, which is non-parametric. They evaluated their model to four different image classification dataset and their model outperformed than other baseline models.\n\n##########################################################################\n\nReasons for score: \n\nEven though this is an interesting setting and the technical solutions presented in the paper look reasonable, the idea seems to be pretty incremental as it stacks multiple existing techniques without many innovations. \n\n##########################################################################\n\nPros: \n\nThe proposed hard attention model finds an optimal location using partial variational auto-encoder. Their attention policy is non-parametric and explainable. They validated their model on four different datasets with qualitatively analyzed results.\n\n##########################################################################\n\nCons: \n\n1. There’s needs for more throughly designed experimental settings with more datasets.\n2. Authors need to perform ablation studies with other form of attention mechanisms. \n\n##########################################################################\n\nQuestions during rebuttal period: \n \nFor me, it's a bit hard to say the proposed methodology is novel. \nAuthors needs to explain why the proposed model is different from pre-existing methodologies regarding attention mechanism.\n \n#########################################################################", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new hard attention model for the image classification as a way to achieve explainability. Two of the reviewers do not find the output of the system interpretable, which is a fatal weakness for a paper on XAI.
R1: The visualization in Fig.5 shows only that the region selected in each timestep indeed has the maximum EIG. But how to interpret the explainability from the glimpse sequence is still confusing. I can hardly perceive the sequence using my knowledge.
R2: However the output of the system is not so appealing either in performance or explainability.
R3: Post-discussion note: For me, it's a bit hard to say the proposed methodology is novel. Authors needs to explain why the proposed model is different from pre-existing methodologies regarding attention mechanism.  
R4: Due to the above, the recommendation is Reject - but the authors are strongly encouraged to do experiments on more challenging data and compare to a newer baseline.

Decision: Reject