Title: PROGRESSIVE ATTENTION NETWORKS FOR VISUAL ATTRIBUTE PREDICTION

Abstract: We propose a novel attention model which can accurately attend to target objects of various scales and shapes in images. The model is trained to gradually suppress irrelevant regions in an input image via a progressive attentive process over multiple layers of a convolutional neural network. The attentive process in each layer determines whether to pass or suppress features at certain spatial locations for use in the next layer. We further employ local contexts to estimate attention probability at each location since it is difficult to infer accurate attention by observing a feature vector from a single location only. The experiments on synthetic and real datasets show that the proposed attention network outperforms traditional attention methods in visual attribute prediction tasks.

1. INTRODUCTION:

Attentive mechanisms often play important roles in modern neural networks (NNs) especially in computer vision tasks. Many visual attention models have been introduced in the previous literature, and they have shown that attaching an attention to NNs can improve the accuracy in various tasks such as image classification (Jaderberg et al., 2015; Ba et al., 2015; Mnih et al., 2014; Larochelle & Hinton, 2010) , image generation (Gregor et al., 2015) , image caption generation (Xu et al., 2015) and visual question answering (Yang et al., 2015; Andreas et al., 2016; Xu & Saenko, 2015) .

There are several motivations for incorporating attentive mechanisms in NNs. One of them is that it is analogous to the perceptual process of human beings. The human visual system concentrates attention to a region of interest instead of processing an entire scene. Likewise, in a neural attention model, we can focus processing only on attended areas of the input image. This benefits us in terms of computational resources; the number of hidden units may be reduced since the hidden activations only need to encode the region with attention (Mnih et al., 2014) .

Another important motivation is that some computer vision tasks, e.g. visual question answering (VQA), require identifying the object for accurate attribute prediction. For example, when the input image contains multiple objects, the task should focus on the object specified by the question. Figure 1 illustrates an example task to predict the color (answer) of a given input number (query). The query specifies a particular object in the input image (number 7 in this example) for answering its attribute (red). To address this type of tasks, the network architecture should incorporate an attentive mechanism either explicitly or implicitly.

One of the most popular attention mechanisms for NNs is the soft attention method (Xu et al., 2015) , which aggregates responses in a feature map weighted by their attention probabilities (see Appendix A for more details). This process results in a single attended feature vector. Since the soft attention method is fully differentiable, the entire network can be trained end-to-end with standard backpropagation. However, it can only model attention to local regions with a certain size depending on the receptive field of the layer chosen for attention. This makes the soft attention method inappropriate for complicated cases, where objects involve significant variations in their scales, and shapes. To overcome this limitation, we propose a novel attention network, referred to as progressive attention network (PAN), which enables precise attention over objects of different scales and shapes by attaching attentive mechanisms to multiple layers within a convolutional neural network (CNN). More specifically, the proposed network forces attention prediction in intermediate feature maps by forwarding the attended feature maps in each layer to the subsequent layers in the CNN. Since a feature to be attended in the current feature map is obtained by combining lower-level features with smaller receptive fields, the network can learn to distill the precise spatial support relevant to the target objects as final attention. The contribution of this work is three-fold:

• A novel attention model (progressive attention network) which can be learned to predict attention matching accurate scale and shape of a target object

• Use of local contexts to improve the stability of the progressive attention model

• Achievement of significant performance improvement over traditional soft and hard attention approaches in query-specific visual attribute prediction tasks

The rest of this paper is organized as follows. We first review related work in Section 2. In Section 3, we describe the proposed model with local context information. We then present our experimental results on several datasets in Section 4 and conclude the paper in Section 5.

2. RELATED WORK:

Attention on Features The most straightforward attention mechanism is a feature based method, which selects a subset of features by explicitly attaching an attention model to NN architectures. The approaches relying on this attention mechanism have improved performance in many tasks (Xu et al., 2015; Yang et al., 2015; Andreas et al., 2016; Xu & Saenko, 2015; Bahdanau et al., 2015; Luong et al., 2015; Weston et al., 2015; Graves et al., 2014) . For example, they have been used to handle sequences of variable lengths in neural machine translation models (Bahdanau et al., 2015; Luong et al., 2015) , speech recognition (Chorowski et al., 2014) and handwriting generation (Graves, 2013) , and manage memory access mechanisms for memory networks (Weston et al., 2015) and neural turing machines (Graves et al., 2014) . When applied to computer vision tasks to resolve reference problems, these models are designed to pay attention to CNN features corresponding to subregions in the input image. Image caption generation and visual question answering are typical examples benefited from this attention mechanism (Xu et al., 2015; Yang et al., 2015; Andreas et al., 2016; Xu & Saenko, 2015) .

Attention by Image Transformation Another stream of attention models is based on image transformations. These approaches transform a regular grid and sample from the input image with the transformed grid whose element corresponds to a location in the input image. Ba et al. (2015) and Mnih et al. (2014) transform an input image with predicted translation parameters (t x and t y ) and a fixed scale factor (ŝ < 1) for image classification or multiple object recognition. Scale factor is also predicted in (Gregor et al., 2015) for image generation, where the network uses Gaussian filters for sampling. Spatial transformer networks (STNs) predict all six parameters of the affine transformation matrix, and even extend it to a projective transformation and a 16-point thin plate spline transformation (Jaderberg et al., 2015) . Because all these transformations used in (Jaderberg et al., 2015) involve scale factors, STNs are capable of dealing with objects in different sizes. However, STN is limited when there are multiple candidate regions for attention. Our model overcomes this problem by formulating attention as progressive filtering on feature maps instead of assuming objects can be roughly aligned by a single spatial transformation.

Multiple Attention Processes There have been several approaches iteratively performing attentive processes to resolve relations between targets. Yang et al. (2015) iteratively attend to images conditioned on the previous attention states for visual question answering as the objects of interest are often not specified explicitly in questions but implicitly in relational expressions about the target objects. Also, Weston et al. (2015) and Graves et al. (2014) incorporate attention mechanisms to memory cells iteratively to retrieve different values stored in the memory. Our proposed model is similar in spirit of iterative attention but aimed at attending to a single target object via operating on multiple CNN layers progressively, i.e., attention information is predicted progressively from feature maps through multiple layers of CNN to capture the fine shapes of the target object.

In (Jaderberg et al., 2015) , the authors also conducted an experiment with a network with multiple transformer layers. However, the attention shapes of STNs are still constrained to the type of transformation regardless of the number of transformers. In contrast, the quality of the attention shapes is improved through progressive attention process in the proposed method. Stollenga et al.

(2014) introduced a deep network which manipulates intermediate features of a fixed classifier through channel-wise attention process. Although the channel-wise attention process is used at multiple layers of the network to manipulate the intermediate feature representations, they never explored spatial attention process. More importantly, this method requires to have an accurate pretrained classifier for the target classes prior to learning attention while pretraining a general query-specific attribute classifier is not trivial. It is also notable that both (Jaderberg et al., 2015) and (Stollenga et al., 2014) target simple classification tasks without queries while we aim to tackle the query-specific attribute prediction task where answers from a single input image can be very different depending on the input query.

Training Attention Models The networks with soft attention are fully differentiable and thus trainable end-to-end by backpropagation. Xu et al. (2015) and Zaremba & Sutskever (2015) introduced a stochastic hard attention, where the network explicitly selects a single feature based on the predicted attention probability map. Because the explicit selection (or sampling) procedure is not differentiable, REINFORCE learning rule (Williams, 1992) , is used to make networks trainable. Transformation based attention models (Ba et al., 2015; Mnih et al., 2014) are mostly trained by REINFORCE learning rule but STN (Jaderberg et al., 2015) proposed a fully differentiable formulation and made it possible to train end-to-end. Compared to these attention networks, the proposed network is also trainable end-to-end by the standard backpropagation without any extra techniques since every operation within the network is differentiable.

3. PROGRESSIVE ATTENTION NETWORKS:

To overcome the limitation of existing attention models in handling variable object scales and shapes, we propose a progressive attention mechanism. In the proposed model, irrelevant features at different scales are suppressed by attention filtering steps in different CNN layers, and computation is focused on the features corresponding to regions of interest. At each attention layer, the model predicts an attention map given the input query and the current feature map via an attention module, and then the attention maps is multiplied to the feature maps channel-wise to obtain attended feature map. In each layer, each attended feature map is then forwarded to the next layer of the CNN for construction of the following feature map, which is illustrated in Figure 2 . This progressive attention process allows us to estimate precise details of attention areas while maintaining deep representations appropriate for high-level inference tasks. Figure 2: Overall procedure of progressive attention. Attentive processes are repeatedly applied to feature maps at multiple layers and the resulting attended feature maps are used as input feature maps for the next convolution layers in CNN. Attention probabilities α l are estimated from feature maps and input query. In the last attention layer, the attended feature maps are aggregated to a single feature vector (by sum pooling) and fed to the final attribute classifier.

3.1. PROGRESSIVE ATTENTIVE PROCESS:

Let f l ∈ R H l ×W l ×C l be an output feature map of a layer l ∈ {0, . . . , L} in CNN with width W l , height H l and C l channels, and f l i,j ∈ R C l be a feature at (i, j) of the feature map f l . In the proposed PAN, an attentive process is applied to multiple layers of CNN and we obtain the attended feature map

EQUATION

Here, the attention probability α l i,j for a feature f l i,j is calculated by

EQUATION

where g l att (•) denotes the attention function with a set of parameters θ l att for layer l, s l i,j is the attention score at (i, j) in layer l, q is the query, and σ(•) is a sigmoid function. The attention probability at each location is independent of others in the same feature map, where a sigmoid function is employed to constrain attention probabilities between 0 and 1. For the last layer of attention, we use a softmax function over the entire spatial region for final aggregation of features.

Unlike the soft attention model (see Appendix A), in the intermediate attention layers, the attended feature map f l is not summed up to generate a single vector representation of the attended regions. Instead, the attended feature map is forwarded to the next layer as an input to compute the next feature map, which is given by

EQUATION

where g l+1 CNN (•) is the next CNN operations parameterized by θ l CNN . This feedforward procedure with attentive processes in CNN is repeated from the input of the CNN, i.e., f 0 = I, until f L is obtained. Then, the attended feature f att is finally retrieved by summing up all the features in the final attended feature map f L as in soft attention, which is given by

EQUATION

The attended feature f att obtained by such process is then used as the input to the visual attribute classifier as illustrated in Figure 2 .

In our models, we place the attention layers to the output of max pooling layers instead of every layer in CNN because the reduction of feature resolution within CNN mainly comes from pooling layers.

In practice" we can also skip the first few pooling layers and only attach the attention module to the outputs of last K pooling layers. In (a), α l i,j is predicted from f l i,j only while its spatially adjacent features are also used to estimate α l i,j in (b).

3.2. MULTI-RESOLUTION ATTENTION ESTIMATION:

In Eq. ( 3), the resolution of attention probability map α l depends on the size of the feature map in the corresponding layer. Due to the nature of a CNN with convolution and pooling layers, the resolution of α l will decrease with the increasing depth of a layer. Since the attentive processes are performed over multiple layers recursively in our framework, it is possible to attend to the regions of specific sizes and shapes. Note that the proposed network can exploit high-level semantics in deep representations for inference without losing attention resolution.

The progressive attention model is still very effective in predicting fine attention shapes as the attention information is aggregated over multiple layers to suppress irrelevant structures at different granularity. In lower layers, features whose receptive fields contain small distractors are suppressed first. Meanwhile, the features from a part of large distractors remain intact but passed to the next layer delaying its suppression. In higher layers, features of these large distractors would get low attention probability as each feature contains information from larger receptive fields allowing the attention module to distinguish whether the feature is from a distractor or the target object. This phenomenon is well demonstrated in the qualitative results in our experiments (Section 4). An additional benefit of progressive attention is that it is more straightforward during inference since it is a pure feedforward network.

3.3. LOCAL CONTEXT:

A basic version of PAN discussed so far predicts an attention probability α l i,j based solely on the feature f l i,j at a single feature map location. We can improve the quality of attention estimation by allowing the attention layers to observe a local context of the target feature. The local context F l i,j of a feature f l i,j is composed of its spatially adjacent features. For example, the local context can be given by F l i,j = {f l s,t |i -δ ≤ s ≤ i + δ, j -δ ≤ t ≤ j + δ} as illustrated in Figure 3 . The attention score is now predicted by the attention network with local context as

EQUATION

In this architecture, the area of the local context is given by the filter size corresponding to the composite operation of convolution followed by pooling in the next layer. The local context does not need to be considered in the last layer of attention since its activations are used to compute the final attended feature map. Local context improves attention prediction as it enables the centroid feature to be compared with surrounding features which makes the estimated attention more discriminative.

3.4. TRAINING PROGRESSIVE ATTENTION NETWORKS:

Training a PAN is as simple as training a soft attention network (Xu et al., 2015) because every operation within the network is differentiable. The entire network is trained end-to-end by the standard backpropagation minimizing the binary cross entropies of the object-specific visual attributes. When we train it from a pretrained CNN, the CNN part should always be fine-tuned together since the intermediate attention maps may change the input distributions of their associated layers in CNN. 

4. EXPERIMENTS:



4.1. MNIST REFERENCE:



Datasets:

We conduct experiments on a synthetic dataset created from MNIST (LeCun et al., 1998) .

The synthetic dataset is referred to as MNIST Reference (MREF; Figure 4a ), where each training example is a triple of an image, a query number and its color label. The task on this dataset is to predict the color of the number identified by a query. Five to nine distinct MNIST numbers with different colors in {green, yellow, white, red, blue} and scales in [0.5, 3.0] are randomly sampled and located in each 100 × 100 image. When coloring numbers, Gaussian noise is added to the reference color value. To simulate more realistic situations, we made two variants of MREF by chainging backgrounds to either distractors (MDIST; Figure 4b ) or natural images (MBG; Figure 4c ). Background images in MDIST are constructed with randomly cropped 5 × 5 patches of MNIST images whereas backgrounds of MBG are filled with natural scene images randomly chosen from the SUN Database (Xiao et al., 2014) . The training, validation and test sets contain 30,000, 10,000 and 10,000 images respectively.

Experimental Settings We implement the proposed network with and without the local context observation referred to as PAN-CTX and PAN, respectively. In addition, soft attention network (SAN), hard attention network (HAN) (Xu et al., 2015) and two variants of spatial transformer network (STN-S and STN-M) (Jaderberg et al., 2015) , are used as baseline models for comparisons. While STN-S is the model with a single transformer layer, STN-M contains multiple transformer layers in the network. We reimplemented SAN and STNs following the descriptions in (Xu et al., 2015) and (Jaderberg et al., 2015) , respectively, and trained HAN by optimizing the marginal log-likelihood loss as it is more accurate and feasible due to small search space in our task. The architecture of image encoding network in SAN and HAN and localization networks in STNs are all identical for fair comparisons. CNN in the proposed network also has the same architecture except for the additional layers for hierarchical attention. The CNN is composed of four stacks of 3 × 3 convolutions with 32 channels (stride 1) followed by a 2 × 2 max pooling layer (stride 2) as illustrated in Figure 5a . We used a single fc layer for classification because the task requires simple color prediction. The attention functions g l att (•) for all models are formed as multi-layer perceptrons with two layers (Figure 5b ). The function takes the concatenation of a query q, which is a one-hot vector representing the target object and a feature vector f l i,j , and outputs an attention score s l i,j . In PAN-CTX, the attention functions of att1, att2 and att3 additionally take the local context F l i,j containing the adjacent features with δ = 2. Every model is trained from scratch.

Results Table 1a presents color prediction accuracy of all compared algorithms. It is obvious that PAN outperforms all the previous approaches with significant margins and PAN-CTX further improves the performance by exploiting the local contexts for attention estimation. While STN-S often fails to predict the correct answers, STN-M learns to predict the color of the target object through multiple transformations and shows comparable performance to PAN in MREF. However, the performance of STN-M dramatically drops as the dataset becomes more complex and realistic, resulting in even lower performance than SAN and HAN. Also, note that STN-S is capable of attending to any region attended by STN-M since both models predict attention regions by estimating an affine transformation. STN-M achieves the improvement by learning multiple transformers from gradients coming from different levels of features. In contrast to those parametric models, the proposed network can predict attention map with more fine-grained shapes capturing the spatial support of the target object better.

To evaluate the scale sensitivity of each model, we divided the test images into five subsets based on target object scales with uniform interval and computed the accuracies of the models. The results are presented in Figure 6a , where SAN and HAN tend to predict the correct answers only in a scale range between 1.0 and 2.0, while their performance is degraded significantly with wild scale changes. STN-M becomes vulnerable to scale variations in more realistic settings. In contrast, PAN and PAN-CTX are robust to scale variations due to their multi-scale attention machanism especially when the local contexts are incorporated.

Unlike STNs whose attention is constrained to rhombic regions, those models based on feature-wise attention maps can produce attention regions adaptive to the shapes of the target object. We evaluate the attention quality of these models using two complementary criteria: true-positive ratio (TPR) and precision-recall (PR) curve. TPR measures how strong attention is given to proper location by computing the ratio of the aggregated attention probability within the desired area (a.k.a., groundtruth segmentation) to the attention probability in the whole image (Table 1b ). PR measures the overlaps between ground-truth segmentations and binarized segmentation predictions constructed with different thresholds (Figure 6b ). Note that the proposed model with the local context observation gives the best results with significant margin compared to all the other methods in terms of both criteria. These results suggest that PAN-CTX constructs more accurate shapes of attended regions than all other attention models.

Figure 7 shows the qualitative results of the proposed method and two baselines on the MBG dataset. The proposed model yields accurate attention regions eventually by gradually augmenting attention and suppressing irrelevant regions in the image. We can observe that the proposed model could maintain the high attention resolution through the progressive attention process. In contrast, the baseline models attend to the target objects only once at the top layer resulting in a coarse attention in size and shape. More qualitative results in these experiments are presented in Appendix C.

4.2. ATTRIBUTE PREDICTION ON VISUAL GENOME:

Dataset Visual Genome (VG) (Krishna et al., 2016) is an image dataset containing several types of annotations: question/answer pairs, image captions, objects, object attributes and object relationship. We formulate the object attribute prediction as a multi-label classification task with reference. Given an input image and a query (i.e., an object category), we predict the binary attributes of individual objects specified by a query. We used 827 object classes and 749 attribute classes that appear more than 100 times. A total of 86,674 images with 667,882 object attribute labels are used for our experiment, and they are split into training, validation and test sets each containing 43,337, 8,667 and 34,670 images. The task is challenging because scales of objects largely vary and the attributes may be associated with very small objects.

Experimental Settings and Results:

We mainly compare our algorithm with SAN and HAN since STNs could not learn a proper attention process on VG. The transformer layers of STNs generated padded images of different sizes and rotations to encode the query vector to fit the query-specific biases. All the networks share the same CNN architecture of VGG-16 network (Simonyan & Zisserman, 2015) , which is pretrained on ImageNet (Deng et al., 2009) and is further fine-tuned on the VG dataset for the attribute prediction. For SAN and HAN, an attention layer is attached to the last pooling layer in VGG-16 while PAN stacks an additional attention layer with the local contexts F l i,j with δ = 2 on top of each of the last three pooling layers in VGG-16. We skip to place attention layers at the first two pooling layers (pool1 and pool2) because the features in those layers are not discriminative enough to filter out.We also test models with object class conditional prior. In these models, the final attended feature is fused with the query once more by a fully connected layer allowing the network to reflect the conditional distribution of the attributes given the query. Refer to Appendix B for more detailed descriptions on the network architectures.

All three models are evaluated in terms of mean average precision (mAP) weighted by the frequencies of the attribute labels in the test set, where the computation of mAP follows PASCAL VOC protocol (Everingham et al., 2010) . The proposed method consistently achieves the best weighted mAP scores in both experimental settings as shown in Table 2 but the gain reduces with object class conditional prior. Table 2 also shows TPR of each model measured with the ground-truth bounding box for evaluating the attention qualities, and the proposed method shows the best TPR. Figure 8 presents the qualitative results of the proposed network and HAN on VG dataset.

Appendices:



A SOFT ATTENTION MODEL:

In this appendix section, we explain the soft attention network which is introduced in (Xu et al., 2015) and used as one of the baseline models in the experiments. Given a feature map, the soft attention network calculates an attention probability map and uses it to compute the attended feature for classification or other tasks. Given a feature map f ∈ R H×W ×C and a query q containing information of where to attend, a soft attention model first obtains an attended feature map f ∈ R H×W ×C , where W is width, H is height, and C is the number of channels. The input feature map f is generally a CNN output of an input image I, which is given by f = CNN(I).

(6)

For each feature f i,j ∈ R C at (i, j) of the feature map f and the query q, the attention probability map denoted by α = [α i,j ] is given by

s i,j = g att (f i,j , q; θ att ) (7) α i,j = softmax i,j (s), 0 ≤ α i,j ≤ 1 (8)

where g att (•) is the attention network parameterized by θ att and s = [s i,j ] is an attention score map.

The attention score map is normalized with softmax to produce attention probabilities α i,j . Note that g att (•) can be any kind of network such as a multilayer perceptron.

Let fi,j ∈ R C be a vector of the attended feature map f at (i, j). Then, the attended feature denoted by f att ∈ R C is computed by a weighted sum of features as

EQUATION

Ideally, the locations in the feature map corresponding to the receptive fields containing an object of interest should have the maximum attention probability while the others have zero probabilities similarly to the hard attention. This statement stands true only if the target object is perfectly aligned with the receptive fields in terms of position and scale. In practice, however, object location and size vary whereas the structure of receptive fields is fixed. Note that there exists the trade-off between the attention resolution and the representation power. If we choose to extract deep and high-level features, we give up high resolution in attention. On the other hand, we need to rely on shallow representations to increase attention resolution. This trade-off limits the performance of existing attention models.

B NETWORK ARCHITECTURES ON VISUAL GENOME:

In PAN, the convolution and pooling layers of VGG-16 network (Simonyan & Zisserman, 2015) , pretrained on ImageNet (Deng et al., 2009) , are used, and three additional attention layers att1, att2 and att3 are stacked on top of the last three pooling layers pool3, pool4 and pool5 respectively as illustrated in Figure 9a . The attention functions of att1 and att2 take the local contexts F l i,j in addition to the query q and the target feature f l i,j to obtain the attention score s l i,j . The size of the local contexts is squared with that of the receptive fields of the next three convolution layers before the next attention by setting δ = 3. Three convolutions same as the next three convolution layers in CNN firstly encode the target feature and the local context, and are initiallized with the same weights as in CNN (Figure 9b ). This embedding is then concatenated with the one-hot query vector and fed to two fully connected layers, one fusing two modalities and the other estimating the attention score. In att3, the attention function takes the concatenation of the query and the target feature and feed it to two fully connected layers (Figure 9c ). The attended feature f att obtained from the last attention layer att3 is finally fed to a classification layer to predict the attributes.

The baseline networks also share the same architecture of CNN of VGG-16 network as in PAN (Figure 9a ). In SAN, the soft attention described in Appendix A is attached to the top of CNN. In HAN, the hard attention (Xu et al., 2015) is attached to the top of CNN instead. The hard attention is conv1_1 (3×3@64) conv1_2 (3×3@64) fc (classification layer) HAttNet pool1 (2×2) conv2_1 (3×3@128) conv2_2 (3×3@128) pool2 (2×2) conv3_1 (3×3@256) conv3_2 (3×3@256) conv3_3 (3×3@256) pool3 (2×2) att1 conv4_1 (3×3@512) conv4_2 (3×3@512) conv4_3 (3×3@512) pool4 (2×2) att2 conv5_1 (3×3@512) conv5_2 (3×3@512) conv5_3 (3×3@512) pool5 implemented to maximize the marginal likelihood directly during training while the original paper maximized the variational lower bound of the marginal likelihood because of the large attention search space. For testing, we also directly calculated the marginal likelihood instead of picking a single prediction with the highest attention probability. This is possible because of relatively small search space of attention in our problem compared to the image captioning where the search space of attention increases exponentially depending on the lengths of sequences. The attention functions in the baselines consist of two fully connected layers taking the concatenation of the query and the target feature as in the attention function of att3 in PAN.

EQUATION

The proposed network and the baselines described above use the query for obtaining the attention probabilities and give us the pure strength of the attention models. However, the target object class, represented by the query, gives much more information than just attetion. It confines possible attributes and filters irrelevant attributes. For these reasons, we additionally experiment on a set of models that incorporate the target object class conditional prior for the attribute prediction. In these models, the query is fused with the attended feature f att by an additional fully connected layer and the fused feature is used as the input of the classification layer. 

:

CONCLUSIONWe proposed a novel hierarchical attention network, which progressively attends to regions of interest through multiple layers of a CNN. As the model is recursively applied to multiple layers of CNN with an inherent feature hierarchy, it accurately predicts regions of interest with variable sizes and shapes. We also incorporate local contexts into our attention network for more robust estimation. The proposed network can be trained end-to-end with standard error backpropagation. We tested the model on both synthetic and real datasets, and demonstrated significant performance improvement over existing attention methods.

Figures and Tables:

Figure 1: An example reference problem (with the query 7 and the answer red) and intermediate attention maps using our progressive attention model. It shows that attention is gradually refined through the network layers for resolving the reference problem. Distracting patterns at smaller scales are suppressed at earlier layers while those at larger scales (e.g. 9) are suppressed at later layers with larger receptive fields. All attended images are independently rescaled for the visualization.

Figure 3: Attention estimation (a) without local context and (b) with local context. In (a), α li,j is predicted from f l i,j only while its spatially adjacent features are also used to estimate α l i,j in (b).

Figure 4: Example of the MREF datasets.

Figure 5: Detailed illustration of network architectures on MNIST Reference experiments.

Attribute prediction accuracies of different models on the test subsets in different scales. The precision-recall curves of object segmentation with attention probability.

Figure 6: Analysis of algorithms on MREF (left), MDIST (middle), and MBG (right).

Figure 7: Qualitative results of SAN, HAN and PAN-CTX. (a) Input images faded by attended feature map (c). (b) Magnitude of activations in feature maps f l i,j before attention: the activations are mapped to original image space by spreading activations to their receptive fields. (c) Magnitude of activations in attended feature maps f l i,j which shows the effect of attention in contrast to (b). (d) Magnitude of activations of the attended feature maps f l i,j in its original resolution of the feature map. For PAN-CTX, only last three attention layers are visualized and attentions of ealier layers are accumulated for visualizing higher attention layers. For HAN, (c) and (d) represent attention probability because attended feature map is not available. Every image except for input image is rescaled into [0, 1] by (x -min)/(max -min).

Figure 8: Visualization of example attentions of HAN and PAN-CTX on VG dataset. Attention maps present magnitude of attended features and red boxes show ground truth bounding boxes of query.

Architecture of the intermediate attention functions g l att (•) in att1 and att2 of PAN.

Architecture of the attention functions of SAN and HAN, and the last attention function of PAN.

Figure 9: Detailed illustration of network architectures on Visual Genome experiments.

Figure 11: More qualitative results of SAN, HAN and PAN-CTX on the MBG dataset.

Figure 12: Two common failure cases of attention models on the MBG dataset. (a) The models attend to a part of a larger structure which resembles the target object. (b) The models are confused by background distractors that are similar to the target object. Although failed, the examples show that the results of PAN-CTX are more visually interpretable (attended to query-like structures).

Performance of attention models on MREF, MDIST, and MBG datasets.

Table content: <table><tr><td colspan="2">(a) Color prediction accuracy [%]</td><td colspan="2">(b) True-positive ratio [%]</td></tr><tr><td>MREF MDIST 39.10 38.32 93.89 85.09 82.94 75.73 81.84 78.49 95.92 91.65 PAN-CTX 98.51 STN-S STN-M SAN HAN PAN 96.02</td><td>MBG 32.27 52.25 53.77 55.84 69.46 85.55</td><td>MREF MDIST 2.34 2.35 13.61 12.56 13.95 13.81 17.39 13.10 PAN-CTX 22.59 Uniform SAN HAN PAN 22.80</td><td>MBG 2.39 6.73 7.64 8.62 11.01</td></tr></table>

Weighted mAP of the attribute prediction and TPR of attentions measured with ground-truth bounding boxes on VG dataset.

Table content: <table><tr><td/><td colspan="2">attention only</td><td colspan="2">w/ prior</td></tr><tr><td/><td>mAP</td><td>TPR</td><td>mAP</td><td>TPR</td></tr><tr><td>SAN</td><td>27.62</td><td>15.01</td><td>31.84</td><td>17.65</td></tr><tr><td>HAN</td><td>27.72</td><td>17.24</td><td>31.93</td><td>19.70</td></tr><tr><td>PAN-CTX</td><td>29.38</td><td>18.01</td><td>32.50</td><td>20.17</td></tr></table>

Peer review (iclr2017):
The paper presents an architecture to incrementally attend to image regions - at multiple layers of a deep CNN. In contrast to most other models, the model does not apply a weighted average pooling in the earlier layers of the network but only in the last layer. Instead, the features are reweighted in each layer with the predicted attention. 1. Contribution of approach: The approach to use attention in this way is to my knowledge novel and interesting. 2. Qualitative results: 2.1. I like the large number of qualitative results; however, I would have wished the focus would have been less on the “number” dataset and more on the Visual Genome dataset. 2.2. The qualitative results for the Genome dataset unfortunately does not provide the predicted attributes. It would be interesting to see e.g. the highest predicted attributes for a given query. So far the results only show the intermediate results. 3. Qualitative results: 3.1. The paper presents results on two datasets, one simulated dataset as well as Visual Genome. On both it shows moderate but significant improvements over related approaches. 3.2. For the visual genome dataset, it would be interesting to include a quantitative evaluation how good the localization performance is of the attention approach. 3.3. It would be interesting to get a more detailed understanding of the model by providing results for different CNN layers where the attention is applied. 4. It would be interesting to see results on more established tasks, e.g. VQA, where the model should similarly apply. In fact, the task on the numbers seems to be identical to the VQA task (input/output), so most/all state-of-the-art VQA approaches should be applicable. Other (minor/discussion points) - Something seems wrong in the last two columns in Figure 11: the query “7” is blue not green. Either the query or the answer seem wrong. - Section 3: “In each layer, the each attended feature map” -> “In each layer, each attended feature map” - I think Appendix A would be clearer if it would be stated that is the attention mechanism used in SAN and which work it is based on. Summary: While the experimental evaluation could be improved with more detailed evaluation, comparisons, and qualitative results, the presented evaluation is sufficient to validate the approach. The approach itself is novel and interesting to my knowledge and speaks for acceptance.